qwen3.5-latex-ocr-finetune-v2

Qwen3.5-0.8B — Fine-tuned for LaTeX OCR (v2, Full-Scale Training)

This is the full-scale successor to bhaskar1707/qwen3.5-latex-ocr-finetune, which was trained on a 30-step demo subset purely to validate the fine-tuning pipeline. This v2 model is trained on the complete unsloth/LaTeX_OCR dataset with a proper train/validation/test split, giving a much more reliable estimate of real-world performance.

This model is a fine-tuned version of unsloth/Qwen3.5-0.8B, a vision-language model, adapted to convert images of handwritten/printed math formulas into LaTeX code.

Fine-tuning was performed using Unsloth with LoRA (parameter-efficient fine-tuning) on the unsloth/LaTeX_OCR dataset.

Model Details

• Base model: unsloth/Qwen3.5-0.8B (vision-language)
• Fine-tuning method: LoRA (r=16, alpha=16, all attention + MLP + vision layers)
• Framework: Unsloth + TRL (SFTTrainer)
• Dataset: unsloth/LaTeX_OCR — full dataset (handwritten/printed math formula images paired with LaTeX transcriptions)
• Task: Image → LaTeX transcription
• Predecessor: bhaskar1707/qwen3.5-latex-ocr-finetune (30-step demo run, small subset)

Dataset Split

Training Setup

Full per-step logs are available in training_loss_history.csv and validation_loss_history.csv.

Evaluation — Before vs. After Fine-tuning

Evaluated on a held-out test split of 200 samples (not seen during training), using Character Error Rate (CER), exact-match accuracy, and BLEU.

Full numeric results: metrics.json, before_after_comparison.csv.

Qualitative Examples

A sample of qualitative before/after transcription examples (image → predicted LaTeX) is available in qualitative_before_after_examples.csv.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import torch

model_id = "bhaskar1707/qwen3.5-latex-ocr-finetune-v2"

model = AutoModelForImageTextToText.from_pretrained(model_id, torch_dtype=torch.float16, trust_remote_code=True).to("cuda")
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

image = Image.open("your_formula_image.png").convert("RGB")
instruction = "Write the LaTeX representation for this image."

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": instruction}]}]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")

output_ids = model.generate(**inputs, max_new_tokens=128, temperature=1.5, min_p=0.1)
latex = processor.batch_decode(output_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(latex)

Or with Unsloth (faster load/inference, requires a CUDA GPU):

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained("bhaskar1707/qwen3.5-latex-ocr-finetune-v2", load_in_4bit=True)
FastVisionModel.for_inference(model)

Limitations

• Performance is best on formulas visually similar to the LaTeX_OCR dataset distribution; handwriting styles or symbols far outside this distribution may transcribe less accurately.
• As with any generative model, occasional hallucinated symbols or malformed LaTeX are possible, especially for long/complex expressions.
• Evaluation test set (200 samples) is drawn from the same distribution as training data; performance on out-of-distribution formula styles is not separately benchmarked.

Citation / Acknowledgements

• Base model and fine-tuning framework: Unsloth AI
• Dataset: unsloth/LaTeX_OCR

Fine-tuned by Bhaskar Pal — GitHub · LinkedIn