qwen2vl-doc-ocr-lora

qwen2vl-doc-ocr-lora

A LoRA adapter for Qwen2-VL-7B-Instruct fine-tuned for document OCR — given a scanned document image, it outputs the extracted text in reading order.

• Developed by: Abhiram1404
• License: apache-2.0
• Finetuned from: unsloth/Qwen2-VL-7B-Instruct-unsloth-bnb-4bit
• Task: image-to-text (OCR) via a fixed instruction prompt: "Extract all the text from this document image, preserving line order."

This qwen2_vl model was trained 2x faster with Unsloth and TRL's SFTTrainer.

Training data

Trained on real scanned documents from RVL-CDIP (38.7 GB tarball, 16 document classes — letter, form, invoice, resume, memo, etc.) paired with real per-image OCR annotations from jordyvl/rvl_cdip_easyocr (EasyOCR word/box outputs, reconstructed into reading-order text lines).

Caveat: RVL-CDIP is built from 1990s tobacco-litigation document scans — real scan artifacts, but era-appropriate layouts (e.g. its "resume" class looks typewriter/dot-matrix-era, not like a modern resume). Ground truth is EasyOCR's own output, not human transcription, so it inherits EasyOCR's error rate on these low-quality scans.

Training procedure

• Method: LoRA via FastVisionModel.get_peft_model, r=16, lora_alpha=16, lora_dropout=0, target_modules="all-linear", adapters on both vision and language layers (attention + MLP).
• Base model: loaded in 4-bit (load_in_4bit=True) with "unsloth" gradient checkpointing.
• Trainer: TRL SFTTrainer with UnslothVisionDataCollator, batch size 1, gradient accumulation 8 (effective batch size 8), 120 steps, learning_rate=2e-4, linear schedule with 5 warmup steps, adamw_8bit, bf16/fp16 mixed precision, max_seq_length=2048.
• Data: ~2,000 training examples / 100 eval examples sampled from a pooled subset of RVL-CDIP images with usable OCR text (≥15 words).

How to use

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "Abhiram1404/qwen2vl-doc-ocr-lora",
    load_in_4bit=True,
)
FastVisionModel.for_inference(model)

instruction = "Extract all the text from this document image, preserving line order."
messages = [{"role": "user", "content": [{"type": "image", "image": image}, {"type": "text", "text": instruction}]}]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer(image, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")

output_ids = model.generate(**inputs, max_new_tokens=512, temperature=0.2, do_sample=True)
print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))