odia-ocr-qwen-finetuned_v3

Original work by Shantipriya Parida — maintained by OdiaGenAIOCR. For the original personal repo (LoRA adapters, checkpoints) see shantipriya/odia-ocr-qwen-finetuned_v3.

---

Odia OCR — Qwen2.5-VL-7B LoRA Fine-tune (v3)

🚀 Live Demo:
Try the model interactively — no setup required.

A LoRA adapter fine-tuned on top of Qwen/Qwen2.5-VL-7B-Instruct for Odia script Optical Character Recognition (OCR).
The model is trained to extract printed and synthetic Odia text from paragraph-level document images.

---

Model Details

LoRA Configuration

• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Rank / Alpha: 16 / 32
• Dropout: 0.05

---

Training Data

Fine-tuned on a merged dataset of 58 720 image–text pairs from two sources:

Images contain paragraph-level Odia text printed in varied fonts, sizes and layouts.

---

Usage

Checkpoint guide:

• Merged model — shantipriya/odia-ocr-qwen-finetuned_v3-merged — fully merged weights, no PEFT required, easiest to use
• checkpoint-1800 — latest LoRA adapter (CER=0.750, Acc=25%)
• checkpoint-1300 — ⭐ best LoRA adapter (CER=0.655, Acc=34.5%)

Option A — Merged Model (Easiest, No PEFT Required)

The fully merged model (base + LoRA-1800 baked in) is available as a standalone model:

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "shantipriya/odia-ocr-qwen-finetuned_v3-merged",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("shantipriya/odia-ocr-qwen-finetuned_v3-merged")
model.eval()

image = Image.open("odia_document.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text",  "text": "Transcribe all the Odia text from this image exactly as it appears."}
    ]
}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=512)

output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=True
)[0]
print(output)

Option B — LoRA Adapter with Checkpoint-1800 (latest)

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info
from PIL import Image
import torch

REPO = "shantipriya/odia-ocr-qwen-finetuned_v3"
CHECKPOINT = "checkpoint-1800"   # latest evaluated checkpoint

# Load base model
base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load LoRA adapter — checkpoint-1800
model = PeftModel.from_pretrained(base, REPO, revision=CHECKPOINT)
processor = AutoProcessor.from_pretrained(REPO, revision=CHECKPOINT)
model.eval()

# Run OCR on an image
image = Image.open("odia_document.png").convert("RGB")

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text",  "text": "Transcribe all the Odia text from this image exactly as it appears."}
    ]
}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=512)

output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)[0]
print(output)

Option C — LoRA Adapter with Checkpoint-1300 (⭐ best accuracy)

To load the best-performing checkpoint instead, replace the two lines above with:

CHECKPOINT = "checkpoint-1300"   # best checkpoint — CER=0.655, Acc=34.5%

model = PeftModel.from_pretrained(base, REPO, revision=CHECKPOINT)
processor = AutoProcessor.from_pretrained(REPO, revision=CHECKPOINT)

---

Training Metrics

Training loss drops sharply as the model adapts to Odia OCR:

⚠️ Overfitting note: Best checkpoint is 1300 (CER=0.655, Acc=34.5%). Performance degrades after step 1300 despite training loss continuing to drop.

Benchmark CER & Accuracy vs Checkpoint

Checkpoints are pushed every 100 training steps.
Accuracy is reported as 1 − CER (character-level).
⚠️ Eval dataset note: Steps 0/100/200 used word-level images (shantipriya/odia-ocr-merged, single words), which yields lower CER. Step 300 used paragraph-level images (OdiaGenAIOCR/synthetic_data, full paragraphs, ~300 chars) — a much harder task. The lower accuracy at step 300 reflects the harder benchmark, not regression. Full paragraph-level evaluation will be the standard going forward.

Per-category Accuracy: Best (ckpt-1300) vs Latest (ckpt-1800)

Accuracy per OCR category — comparing the best checkpoint (ckpt-1300) against the latest checkpoint (ckpt-1800).
Handwritten and Scene Text categories show the strongest performance; Book and Newspaper remain the hardest.

---

🖼️ Sample Outputs — Checkpoint 1300 (⭐ Best)

Six real examples from the Iftesha/odia-ocr-benchmark dataset (out-of-domain).
Quality: ✅ Good (CER < 0.15) · 🔶 Mixed (CER 0.15–0.65) · ❌ Poor (CER > 0.65)

---

✅ Sample 1 — Motivational Quote (Digital) · Acc: 98.2% · CER: 0.018

Full text reproduced correctly — only Odia danda (।) replaced by period (.) and minor spacing differences.

---

✅ Sample 2 — Poem "ମାଆ" / Mother (Digital) · Acc: 92.9% · CER: 0.071

All 8 stanzas captured. Minor errors: quoted title dropped, ଅମାବାସ୍ଯାର → ଅମାଦାସ୍ଯାର, linebreaks flattened into a single paragraph.

---

✅ Sample 3 — Poem "ସମଯର ଖେଳ" / Game of Time (Digital) · Acc: 91.0% · CER: 0.090

All 4 stanzas transcribed. Title ଖେଳ → ଗୋଳ, ଧୀରେଧୀରେ → ଧୂରେଧୂରେ; complex conjunct characters partially misread.

---

🔶 Sample 4 — News Digest Graphic (Digital) · Acc: 76.1% · CER: 0.239

Recognises 7-headline bulletin structure. Styled/overlapping text causes errors: ଆଜି → ଆନ୍ଦ୍ର, ସ୍କୁଲ → ସୁଲ, ସୁଭଦ୍ରା → ସୁତଦ୍ଧା.

---

🔶 Sample 5 — Book Preface "ମୁଖବନ୍ଧ" (Book) · Acc: 73.4% · CER: 0.266

First 77% accurately read. Complex Sanskrit compound words partially misread. Output truncated before final 2 sentences.

---

❌ Sample 6 — Newspaper Story "ସ୍ନେହର ଢୋରି" (Newspaper) · Acc: 34.2% · CER: 0.658

Severe truncation: output only 40% of full text. Multi-column newspaper layout causes reading order errors. ଢୋରି → ତୋରି, ରିଙ୍ ମାଷ୍ଟର → ରିଙ୍ ମାଞ୍ଚର.

---

🖼️ Sample Outputs — Checkpoint 1300 (⭐ Best)

Six real examples from the Iftesha/odia-ocr-benchmark dataset (out-of-domain).
Quality: ✅ Good (CER < 0.15) · 🔶 Mixed (CER 0.15–0.65) · ❌ Poor (CER > 0.65)

---

✅ Sample 1 — Motivational Quote (Digital) · Acc: 98.2% · CER: 0.018

Full text reproduced correctly — only Odia danda (।) replaced by period (.) and minor spacing differences.

---

✅ Sample 2 — Poem "ମାଆ" / Mother (Digital) · Acc: 92.9% · CER: 0.071

All 8 stanzas captured. Minor errors: quoted title dropped, ଅମାବାସ୍ଯାର → ଅମାଦାସ୍ଯାର, linebreaks flattened into a single paragraph.

---

✅ Sample 3 — Poem "ସମଯର ଖେଳ" / Game of Time (Digital) · Acc: 91.0% · CER: 0.090

All 4 stanzas transcribed. Title ଖେଳ → ଗୋଳ, ଧୀରେଧୀରେ → ଧୂରେଧୂରେ; complex conjunct characters partially misread.

---

🔶 Sample 4 — News Digest Graphic (Digital) · Acc: 76.1% · CER: 0.239

Recognises 7-headline bulletin structure. Styled/overlapping text causes errors: ଆଜି → ଆନ୍ଦ୍ର, ସ୍କୁଲ → ସୁଲ, ସୁଭଦ୍ରା → ସୁତଦ୍ଧା.

---

🔶 Sample 5 — Book Preface "ମୁଖବନ୍ଧ" (Book) · Acc: 73.4% · CER: 0.266

First 77% accurately read. Complex Sanskrit compound words partially misread. Output truncated before final 2 sentences.

---

❌ Sample 6 — Newspaper Story "ସ୍ନେହର ଢୋରି" (Newspaper) · Acc: 34.2% · CER: 0.658

Severe truncation: output only 40% of full text. Multi-column newspaper layout causes reading order errors. ଢୋରି → ତୋରି, ରିଙ୍ ମାଷ୍ଟର → ରିଙ୍ ମାଞ୍ଚର.

---

⭐ Best Checkpoint: 1300 (CER=0.655, Acc=34.5%)

Checkpoint-1300 is the best performing checkpoint. Use this for inference:
shantipriya/odia-ocr-qwen-finetuned_v3 — load with revision="checkpoint-1300"

Checkpoint-1800 Benchmark Results (151 samples — Iftesha/odia-ocr-benchmark)

Latest eval at checkpoint-1800 (note: ckpt-1300 remains best overall):

Benchmark: Iftesha/odia-ocr-benchmark
Checkpoint-1800 results (CER=0.750). History: ckpt-1700 CER=0.912, ckpt-1600 CER=0.758, ckpt-1500 CER=0.690, ckpt-1300 CER=0.655 (best), ckpt-900 CER=0.804.
⭐ Recommended checkpoint for inference: ckpt-1300 (34.5% accuracy).

Sample Inferences — Best Checkpoint 1300 (CER=0.655, Acc=34.5%) — 5 per category

Checkpoint-1300 is the best checkpoint overall — lowest CER and highest accuracy across all categories. Each row shows the original image, ground truth text, and model prediction. Quality icons: ✅ Good (CER < 0.15) · 🔶 Mixed (CER 0.15–0.65) · ❌ Bad (CER > 0.65)

Scene Text

Handwritten

Digital

Book

Newspaper

Printed

Evaluated on Iftesha/odia-ocr-benchmark — out-of-domain from training data.
Best performance: handwritten and scene_text categories.
⭐ Use checkpoint-1300 for best inference results.

Sample Inferences — Latest Checkpoint 1800 (CER=0.750, Acc=24.9%) — 5 per category

Latest evaluated checkpoint (step 1800/3000). Note: ckpt-1300 still gives best overall results.

Scene Text

Handwritten

Digital

Book

Newspaper

Printed

In-Domain Long Paragraphs (OdiaGenAIOCR/synthetic_data)

Evaluated on 3 long paragraph samples (>200 chars) at checkpoint-300:

⚠️ These are paragraph-level newspaper/synthetic images — a significantly harder task than word-level OCR. Model is at step 300/3000 (~10% through training).

---

Out-of-Domain Benchmark — Checkpoint 300 (Iftesha/odia-ocr-benchmark — 151 samples)

† CER > 1 on printed indicates hallucination — model generates longer output than the ground truth.

---

Intended Use

• Primary use: OCR on printed Odia text in document or paragraph images
• Language: Odia (ISO 639-1: or) — one of the 22 scheduled languages of India
• Not intended for: handwritten Odia, non-Odia scripts, or real-time edge deployment without quantization

---

Limitations

• Model is actively training; checkpoints before step 3000 will under-perform
• Performance on heavily degraded or handwritten images has not been evaluated
• No RLHF / DPO alignment — outputs are raw OCR transcriptions

---

Project

This model is part of the OdiaGenAI initiative to build open-source AI tools for the Odia language.

• Organization: OdiaGenAI
• Datasets: OdiaGenAIOCR/synthetic_data
• Author: shantipriya

---

Citation

If you use this model, please cite:

@misc{odia-ocr-qwen-v3,
  author    = {Shantipriya Parida},
  title     = {Odia OCR Qwen2.5-VL LoRA Fine-tune v3},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/shantipriya/odia-ocr-qwen-finetuned_v3}
}

---

License

This project is licensed under the Creative Commons Attribution–NonCommercial–ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

For full license details, see: https://creativecommons.org/licenses/by-nc-sa/4.0/