qwen2.5-vl-3b-mjsynth-ocr-lora

Model Card for phucdu/qwen2.5-vl-3b-mjsynth-ocr-lora

This model is a LoRA fine-tuned version of Qwen/Qwen2.5-VL-3B-Instruct specifically optimized for Optical Character Recognition (OCR) tasks using the MJSynth dataset.

Model Details

Model Description

This model utilizes Parameter-Efficient Fine-Tuning (PEFT) via LoRA and 4-bit quantization to adapt the base Qwen2.5-VL-3B-Instruct model for highly accurate text recognition from images. It achieves significant improvements in Character Error Rate (CER) and Exact Match accuracy compared to the base model.

• Developed by: phucdu
• Model type: Vision-Language Model (VLM) / LoRA Adapter
• Language(s) (NLP): English
• License: Apache 2.0 (inheriting from Qwen)
• Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Extracting and reading text from images (Optical Character Recognition - OCR). It is particularly tuned on single-word crops but generalizes well to standard text reading tasks.

How to Get Started with the Model

(Mục này đã được để trống theo yêu cầu của tác giả - Không chứa code tải mô hình).

Training Details

Training Data

A subset of priyank-m/MJSynth_text_recognition.

• Pool size: 8000 examples
• Split: 6400 Train / 800 Validation / 800 Test.

Training Procedure

Training Hyperparameters

• Training regime: LoRA with 4-bit quantization (nf4, double quant) and bfloat16 compute dtype.
• LoRA Rank (r): 16
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Target Modules: down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj
• Epochs: 3
• Batch Size: 16 (Train) / 8 (Eval)
• Learning Rate: 1e-4
• LR Scheduler: Cosine
• Warmup Ratio: 0.03
• Weight Decay: 0.0

Speeds, Sizes, Times

• Hardware: 1x NVIDIA L4 (24GB) GPU on Google Colab Pro.
• Training Time: ~35 minutes (2115 seconds for 3 epochs).

Evaluation

Testing Data, Factors & Metrics

• Testing Data: 800 held-out samples from the MJSynth dataset.
• Metrics: Character Error Rate (CER), Word Error Rate (WER), Exact Match Accuracy.

Results

Comparing Base Model vs. Fine-tuned Model on the test set:

• CER: 1.3056 ➔ 0.0842 (93.5% relative improvement)
• WER: 2.3462 ➔ 0.1437 (93.8% relative improvement)
• Exact Match Accuracy: 62.37% ➔ 85.62% (37.2% relative improvement)
• Avg Inference Latency: ~0.768 sec/image

Summary

The LoRA fine-tuning process significantly enhanced the model's OCR capabilities on the target dataset, bringing the exact match accuracy up to over 85% while massively dropping both CER and WER.