donut-im2latex

donut-latex-full

Overview

This model is an OCR-free image-to-LaTeX model. It takes a rendered image of a mathematical formula as input and directly produces the corresponding LaTeX markup, without a separate text-detection or OCR stage. It was fine-tuned from naver-clova-ix/donut-base on the yuntian-deng/im2latex-100k dataset.

Architecture

Donut is a Vision-Encoder-Decoder model: a Swin Transformer vision encoder reads the input image and a BART-style autoregressive text decoder generates the target sequence. A dedicated task-start token prompts the decoder to emit LaTeX.

Dataset

• Name: yuntian-deng/im2latex-100k
• Task: rendered formula image -> LaTeX string
• Splits: train / validation / test

Training procedure

The model was trained with mixed precision, gradient accumulation, gradient clipping and gradient checkpointing. The best checkpoint was selected on the validation CER with early stopping.

Hyperparameters

• Fine-tuning mode: Full
• Image size (H x W): 320 x 896
• Max target length: 512
• Epochs: 3
• Train batch size: 4
• Gradient accumulation: 4
• Learning rate: 3e-05
• Optimizer: adamw
• LR scheduler: cosine
• Mixed precision: bf16
• Seed: 42

Hardware

• Accelerator: NVIDIA GeForce RTX 5060 Ti (7.96 GB)
• Platform: Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39

Software versions

• torch: 2.10.0+cu128
• transformers: 5.14.1
• datasets: 4.3.0
• accelerate: 1.12.0
• peft: 0.18.1

Evaluation

Metrics computed on the validation split:

Results

• Best epoch: 3
• Best cer: 0.0558

Intended use

Converting images of typeset mathematical formulas (for example, screenshots of rendered LaTeX) into editable LaTeX source. Suitable for document digitization, accessibility tooling and dataset construction.

Out-of-scope use

• Handwritten mathematics (the training data is rendered/typeset).
• Full-page document layout or natural-scene text recognition.
• Any safety-critical use without human verification of the output.

Limitations and failure cases

• Very long or deeply nested expressions may be truncated or mis-nested.
• Rare symbols underrepresented in the training data may be mispredicted.
• Low-resolution, noisy or heavily styled renderings degrade accuracy.
• Predictions are not guaranteed to compile; downstream validation is recommended.

Bias and ethical considerations

The dataset consists of machine-rendered formulas from scientific papers and is limited to the notation and symbol distribution of that corpus. The model may underperform on notation conventions outside this distribution. Outputs should be reviewed before use in academic or production settings.

Future work

• Extend to handwritten formula recognition.
• Add constrained decoding to improve LaTeX compilability.
• Explore larger input resolutions and beam search tuning.

Example usage

import torch
from PIL import Image
from transformers import DonutProcessor, VisionEncoderDecoderModel

model_id = "thealper2/donut-im2latex"
processor = DonutProcessor.from_pretrained(model_id)
model = VisionEncoderDecoderModel.from_pretrained(model_id)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

image = Image.open("formula.png").convert("RGB")
pixel_values = processor.image_processor(image, return_tensors="pt").pixel_values.to(device)

decoder_input_ids = torch.tensor(
    [[model.config.decoder_start_token_id]], device=device
)
generated = model.generate(
    pixel_values=pixel_values,
    decoder_input_ids=decoder_input_ids,
    max_length=model.generation_config.max_length,
    num_beams=4,
)
latex = processor.tokenizer.batch_decode(generated, skip_special_tokens=True)[0]
print(latex)

Citation

@misc{donut_latex,
  title  = {Donut-LaTeX: OCR-free image-to-LaTeX with Donut},
  author = {donut-latex authors},
  year   = {2024},
  note   = {Fine-tuned from naver-clova-ix/donut-base on yuntian-deng/im2latex-100k}
}

@inproceedings{kim2022donut,
  title     = {OCR-free Document Understanding Transformer},
  author    = {Kim, Geewook and Hong, Teakgyu and Yim, Moonbin and others},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2022}
}

License

Released under the Apache-2.0 license (placeholder -- confirm the license of any derived data and base model before publishing).