Mubsir-Qwen-2B-VL

Mubsir-Qwen-2B-VL: Arabic Handwritten Text Recognition (HTR)

<div align="center"> <img src="logo.png" alt="Arabic Handwriting VLM Logo" width="400" /> </div>

This repository contains a vision-language model fine-tuned specifically for Arabic Handwritten Text Recognition (HTR). Built on the Qwen2-VL-2B-Instruct architecture, it is designed to accurately transcribe complex, unstructured Arabic handwriting into digital text.

A standalone, unmerged parameter-efficient LoRA adapter for this model is also available at Hatim2221/Mubsir-vl-arabic-htr-adapter.

Model Details

• Base Architecture: Qwen/Qwen2-VL-2B-Instruct
• Task: Optical Character Recognition (OCR) / Handwritten Text Recognition (HTR) for Arabic text.
• Precision: torch.float16. The trained LoRA weights were merged into the base model entirely in 16-bit precision. This avoids the mathematical rounding errors and accuracy degradation commonly associated with merging weights directly into 4-bit quantized base models.
• Parameters: 2 Billion.

Training Data

The model was fine-tuned using the KHATT (KFUPM Handwritten Arabic TexT) dataset, a comprehensive collection of diverse Arabic handwritten text lines designed for training high-accuracy recognition systems.

Training Infrastructure & Methodology

• Hardware: 2x NVIDIA T4 GPUs (Kaggle cluster).
• Methodology: Parameter-Efficient Fine-Tuning (PEFT) via QLoRA.
• VRAM Optimization: The training pipeline utilized PyTorch expandable_segments for strict memory management, coupled with dynamic image resolution capping and tailored batch sizing to bypass standard Out-Of-Memory (OOM) failures during multimodal processing.

Use Cases

Example 1: Arabic Handwriting Transcription

Input: <div align="center"> <img src="image1.jpeg" alt="image" width="500" height="200" />

</div> **Prompt:** `Transcribe this Arabic handwriting:` **VLM Output:** > The text on the paper is in Arabic. It reads: "من جبر وجر وصا زرع دمر"

Usage and Inference

To run this model, ensure you have the required Hugging Face libraries installed, including the Qwen vision utilities.

Installation

pip install -q -U transformers accelerate qwen-vl-utils torchvision torchao

Inference code

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model_id = "Hatim2221/Mubsir-Qwen-2B-VL"

# Load the model and processor in native FP16 precision
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

# Define the multimodal input payload
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path_to_your_arabic_handwriting_image.jpg"}, 
            {"type": "text", "text": "Transcribe this Arabic handwriting:"}
        ]
    }
]

# Process text and vision inputs using the Qwen utility
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt"
).to("cuda")

# Generate the transcription
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=128)

# Isolate the generated output from the input tokens
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]

output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print("Transcription Result:")
print(output_text[0])