granite-docling-258M-untied

granite-docling-258m

Model Summary: Granite Docling is a multimodal Image-Text-to-Text model engineered for efficient document conversion. It preserves the core features of Docling while maintaining seamless integration with DoclingDocuments to ensure full compatibility.

Granite Docling 258M builds upon the IDEFICS3 architecture, but introduces two key modifications: it replaces the vision encoder with siglip2-base-patch16-512 and substitutes the language model with a Granite 165M LLM.

• Developed by: IBM Research
• Model type: Multi-modal model (image+text-to-text)
• Language(s): English (NLP)
• License: Apache 2.0
• Release Date: September 17, 2025

Granite-docling-258M is fully integrated into the Docling pipelines, carrying over existing features while introducing a number of powerful new features, including:

• 🔢 Enhanced Equation Recognition: More accurate detection and formatting of mathematical formulas
• 🧩 Flexible Inference Modes: Choose between full-page inference, bbox-guided region inference
• 🧘 Improved Stability: Tends to avoid infinite loops more effectively
• 🧮 Enhanceed Inline Equations: Better inline math recognition
• 🧾 Document Element QA: Answer questions about a document’s structure such as the presence and order of document elements
• 🌍 Japanese, Arabic and Chinese support (experimental)

Evaluations

<table> <thead> <tr> <th></th> <th><b>smoldocling-256m-preview</b></th> <th><b>granite-docling-258m</b></th> </tr> </thead> <tbody> <tr><td colspan="3"><b>Layout</b></td></tr> <tr><td>MAP ↑</td><td>0.21</td><td><b>0.28</b></td></tr> <tr><td>F1 ↑</td><td>0.79</td><td><b>0.85</b></td></tr> <tr><td>Precision ↑</td><td>0.86</td><td><b>0.87</b></td></tr> <tr><td>Recall ↑</td><td>0.82</td><td><b>0.89</b></td></tr> <tr><td colspan="3"><b>Full Page OCR</b></td></tr> <tr><td>Edit-distance ↓</td><td>0.48 (0.46)</td><td><b>0.46</b> (<b>0.44</b>)</td></tr> <tr><td>F1 ↑</td><td><b>0.80</b> (0.76)</td><td>0.75 (<b>0.78</b>)</td></tr> <tr><td>Precision ↑</td><td><b>0.89</b> (0.85)</td><td>0.81 (0.85)</td></tr> <tr><td>Recall ↑</td><td><b>0.79</b> (0.74)</td><td>0.73 (<b>0.77</b>)</td></tr> <tr><td>BLEU ↑</td><td><b>0.58</b> (0.54)</td><td>0.56 (<b>0.59</b>)</td></tr> <tr><td>Meteor ↑</td><td>0.67 (0.67)</td><td>0.67 (<b>0.70</b>)</td></tr> <tr><td colspan="3"><b>Code Recognition</b></td></tr> <tr><td>Edit-distance ↓</td><td>0.114</td><td><b>0.013</b></td></tr> <tr><td>F1 ↑</td><td>0.915</td><td><b>0.988</b></td></tr> <tr><td>Precision ↑</td><td>0.94</td><td><b>0.99</b></td></tr> <tr><td>Recall ↑</td><td>0.909</td><td><b>0.988</b></td></tr> <tr><td>BLEU ↑</td><td>0.875</td><td><b>0.983</b></td></tr> <tr><td>Meteor ↑</td><td>0.889</td><td><b>0.986</b></td></tr> <tr><td colspan="3"><b>Equation Recognition</b></td></tr> <tr><td>Edit-distance ↓</td><td>0.119</td><td><b>0.073</b></td></tr> <tr><td>F1 ↑</td><td>0.947</td><td><b>0.968</b></td></tr> <tr><td>Precision ↑</td><td>0.959</td><td><b>0.968</b></td></tr> <tr><td>Recall ↑</td><td>0.941</td><td><b>0.969</b></td></tr> <tr><td>BLEU ↑</td><td>0.824</td><td><b>0.893</b></td></tr> <tr><td>Meteor ↑</td><td>0.878</td><td><b>0.927</b></td></tr> <tr><td colspan="3"><b>Table Recognition (FinTabNet 150dpi)</b></td></tr> <tr><td>TEDS (structure) ↑</td><td>0.82</td><td><b>0.97</b></td></tr> <tr><td>TEDS (w/content) ↑</td><td>0.76</td><td><b>0.96</b></td></tr> <tr><td colspan="3"><b>Other Benchmarks</b></td></tr> <tr><td>MMStar ↑</td><td>0.17</td><td><b>0.3</b></td></tr> <tr><td>OCRBench ↑</td><td>338</td><td><b>500</b></td></tr> </table>

Getting started

You can use transformers, vllm, or onnx to perform inference, and Docling to convert results to variety of output formats (md, html, etc.):

<details> <summary>📄 Single page image inference using Tranformers 🤖</summary>

# Prerequisites:
# pip install torch
# pip install docling_core
# pip install transformers

import torch
from docling_core.types.doc import DoclingDocument
from docling_core.types.doc.document import DocTagsDocument
from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image
from pathlib import Path

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# Load images
image = load_image("https://upload.wikimedia.org/wikipedia/commons/7/76/GazettedeFrance.jpg")

# Initialize processor and model
processor = AutoProcessor.from_pretrained("ibm-granite/granite-docling-258M")
model = AutoModelForVision2Seq.from_pretrained(
    "ibm-granite/granite-docling-258M",
    torch_dtype=torch.bfloat16,
    _attn_implementation="flash_attention_2" if DEVICE == "cuda" else "eager",
).to(DEVICE)

# Create input messages
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Convert this page to docling."}
        ]
    },
]

# Prepare inputs
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
inputs = inputs.to(DEVICE)

# Generate outputs
generated_ids = model.generate(**inputs, max_new_tokens=8192)
prompt_length = inputs.input_ids.shape[1]
trimmed_generated_ids = generated_ids[:, prompt_length:]
doctags = processor.batch_decode(
    trimmed_generated_ids,
    skip_special_tokens=False,
)[0].lstrip()

# Populate document
doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([doctags], [image])
print(doctags)
# create a docling document
doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="Document")

# export as any format
# HTML
# Path("Out/").mkdir(parents=True, exist_ok=True)
# output_path_html = Path("Out/") / "example.html"
# doc.save_as_html(output_path_html)
# MD
print(doc.export_to_markdown())

</details> <details> <summary> 🚀 Fast Batch Inference Using VLLM</summary>

# Prerequisites:
# pip install vllm
# pip install docling_core
# place page images you want to convert into "img/" dir

import time
import os
from vllm import LLM, SamplingParams
from PIL import Image
from docling_core.types.doc import DoclingDocument
from docling_core.types.doc.document import DocTagsDocument
from pathlib import Path

# Configuration
MODEL_PATH = "ibm-granite/granite-docling-258M"
IMAGE_DIR = "img/"  # Place your page images here
OUTPUT_DIR = "out/"
PROMPT_TEXT = "Convert page to docling."

# Ensure output directory exists
os.makedirs(OUTPUT_DIR, exist_ok=True)

# Initialize LLM
llm = LLM(model=MODEL_PATH, limit_mm_per_prompt={"image": 1})

sampling_params = SamplingParams(
    temperature=0.0,
    max_tokens=8192
)

# Load and prepare all images and prompts up front
batched_inputs = []
image_names = []

for img_file in sorted(os.listdir(IMAGE_DIR)):
    if img_file.lower().endswith((".png", ".jpg", ".jpeg")):
        img_path = os.path.join(IMAGE_DIR, img_file)
        with Image.open(img_path) as im:
            image = im.convert("RGB")

        prompt = (
            f"<|start_of_role|>user<|end_of_role|><image>{PROMPT_TEXT}<|end_of_text|>\n"
            f"<|start_of_role|>assistant<|end_of_role|>"
        )
        batched_inputs.append({"prompt": prompt, "multi_modal_data": {"image": image}})
        image_names.append(os.path.splitext(img_file)[0])

# Run batch inference
start_time = time.time()
outputs = llm.generate(batched_inputs, sampling_params=sampling_params)

# Postprocess all results
for img_fn, output, input_data in zip(image_names, outputs, batched_inputs):
    doctags = output.outputs[0].text
    output_path_dt = Path(OUTPUT_DIR) / f"{img_fn}.dt"
    output_path_md = Path(OUTPUT_DIR) / f"{img_fn}.md"

    with open(output_path_dt, "w", encoding="utf-8") as f:
        f.write(doctags)

    # Convert to DoclingDocument and save markdown
    doctags_doc = DocTagsDocument.from_doctags_and_image_pairs([doctags], [input_data["multi_modal_data"]["image"]])
    doc = DoclingDocument.load_from_doctags(doctags_doc, document_name="Document")
    doc.save_as_markdown(output_path_md)

print(f"Total time: {time.time() - start_time:.2f} sec")

</details>

💻 Local inference on Apple Silicon with MLX: see here

Supported Instructions

<table> <tr> <th>Description</th> <th>Instruction</th> <th>Short Instruction</th> </tr> <tr> <td><b>Full conversion</b></td> <td>Convert this page to docling.</td> <td>-</td> </tr> <tr> <td><b>Chart</b></td> <td>Convert chart to table.</td> <td><code>&lt;chart&gt;</code></td> </tr> <tr> <td><b>Formula</b></td> <td>Convert formula to LaTeX.</td> <td><code>&lt;formula&gt;</code></td> </tr> <tr> <td><b>Code</b></td> <td>Convert code to text.</td> <td><code>&lt;code&gt;</code></td> </tr> <tr> <td><b>Table</b></td> <td>Convert table to OTSL. (<a href="https://arxiv.org/pdf/2305.03393">Lysak et al., 2023</a>)</td> <td><code>&lt;otsl&gt;</code></td> </tr> <tr> <td rowspan="4"><b>Actions and Pipelines</b></td> <td>OCR the text in a specific location: &lt;loc_155&gt;&lt;loc_233&gt;&lt;loc_206&gt;&lt;loc_237&gt;</td> <td>-</td> </tr> <tr> <td>Identify element at: &lt;loc_247&gt;&lt;loc_482&gt;&lt;10c_252&gt;&lt;loc_486&gt;</td> <td>-</td> </tr> <tr> <td>Find all 'text' elements on the page, retrieve all section headers.</td> <td>-</td> </tr> <tr> <td>Detect footer elements on the page.</td> <td>-</td> </tr> </table>

Model Architecture:

The architecture of granite-docling-258m consists of the following components:

(1) Vision encoder: siglip2-base-patch16-512.

(2) Vision-language connector: pixel shuffle projector (as in idefics3)

(3) Large language model: Granite 165M.

We built upon Idefics3 to train our model. We incorporated DocTags into our LLM’s supervised fine-tuning (SFT) data to help the model become familiar with the format, enabling faster convergence and mitigating issues previously observed with SmolDocling. The model was trained using the nanoVLM framework, which provides a lightweight and efficient training setup for vision-language models

Training Data

Our training corpus consists of two principal sources: (1) publicly available datasets and (2) internally constructed synthetic datasets designed to elicit specific document understanding capabilities.

In particular, we incorporate:

• SynthCodeNet — a large-scale collection of synthetically rendered code snippets spanning over 50 programming languages
• SynthFormulaNet — a dataset of synthetic mathematical expressions paired with ground-truth LaTeX representations
• SynthChartNet — synthetic chart images annotated with structured table outputs
• DoclingMatix — a curated corpus of real-world document pages sampled from diverse domains

Infrastructure:

We train granite-docling-258m using IBM's super computing cluster, Blue Vela, which is outfitted with NVIDIA H100 GPUs. This cluster provides a scalable and efficient infrastructure for training our models over thousands of GPUs.

Resources

• ⭐️ Learn about the latest updates with Docling: https://docling-project.github.io/docling/#features
• 🚀 Get started with Docling concepts, integrations and tutorials: https://docling-project.github.io/docling/getting_started/
• 💡 Learn about the latest Granite learning resources: https://ibm.biz/granite-learning-resources