qwen2-vl-id-card-lora

Qwen2-VL-2B-Instruct LoRA for Synthetic ID Card OCR & Document Understanding

This repository contains a Low-Rank Adaptation (LoRA) adapter for Qwen2-VL-2B-Instruct fine-tuned to extract key structural fields from Employee Identification Cards. It was trained and evaluated on programmatically generated synthetic cards to ensure data privacy (PII compliance) and robustness against scanning distortions.

Model Details

• Base Model: Qwen/Qwen2-VL-2B-Instruct
• Adapter Type: PEFT / LoRA
• Language(s): English (text & document layout understanding)
• Task: Key-Value Field Extraction / OCR Document QA
• Fine-Tuning Method: QLoRA (4-bit quantized base weights + 16-bit LoRA adapter projection layers)

Intended Fields to Extract:

• Company Name (e.g., APEX ENTERPRISES)
• Full Name (e.g., STEVEN RODRIGUEZ)
• Employee ID (e.g., NEX-5239-Y)
• Role/Department (e.g., Project Manager)
• Date of Birth (e.g., 2000-11-19)
• Expiry Date (e.g., 2035-01-25)

---

Evaluation & Robustness Benchmarks

The model was evaluated on a dedicated, completely unseen Test Set of 50 synthetic ID cards across four distinct environmental noise scenarios (Clean, Gaussian Blur, 10-degree Tilt/Rotation, and Low Light).

Below are the actual measured results comparing the pre-trained Base Model against the Fine-Tuned Model:

Key Observations:

• OCR Format Learning: The Base Model struggled significantly with the layout formatting (represented by high baseline Character Error Rates), whereas the Fine-Tuned Model achieved a perfect 0.00% CER on Clean, Rotated, and Low Light images.
• Robustness: Under harsh rotation (10-degree scan tilt) and extreme low light (75% luminance drop), the fine-tuned model maintained a 100% exact match rate across all fields, proving layout generalization.

---

Training Details

Training Data

• Dataset: 500 programmatically generated corporate ID cards with random names, roles, company logos, and barcodes.
• Split: 400 Train / 50 Validation / 50 Test.

Training Procedure & Hyperparameters

• Epochs: 3
• Batch Size: 2 (Gradient Accumulation Steps = 8, Effective Batch Size = 16)
• Optimizer: paged_adamw_8bit (memory pages optimization)
• Precision: fp16 (mixed precision training)
• Learning Rate: 2e-4 (cosine scheduler decay)
• LoRA configuration:
  • Rank (r): 16
  • Alpha: 32
  • Dropout: 0.05
  • Target Modules: ["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]

---

How to Get Started with the Model

You can load this model along with its adapter using the following snippet:

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel

model_id = "Qwen/Qwen2-VL-2B-Instruct"
adapter_id = "YOUR_HF_USERNAME/qwen2-vl-id-card-lora" # Replace with your repo identifier

# Load processor
processor = AutoProcessor.from_pretrained(model_id)

# Load base model (quantized or full precision)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)