Mubsir-vl-arabic-htr-adapter

Mubsir — Arabic Handwritten Text Recognition (HTR) Adapter

Mubsir (مُبصِر — "one who sees clearly") is a QLoRA adapter that fine-tunes Qwen2-VL-2B-Instruct for Arabic handwritten text recognition. Given an image of unstructured Arabic handwriting, the model outputs an accurate digital transcript.

This repository contains the standalone LoRA adapter weights only. For a ready-to-use, fully merged 16-bit model, see Hatim2221/Mubsir-Qwen-2B-VL.

Model Details

Training Setup

• Hardware: Kaggle Notebooks, 2× NVIDIA T4 GPUs
• Frameworks: PyTorch, transformers, peft, accelerate, qwen-vl-utils
• Trainer: Native Hugging Face Trainer (used in place of SFTTrainer to avoid library versioning conflicts in the Kaggle environment)
• Memory management: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, capped dynamic image resolution, and reduced batch sizes to fit multimodal fine-tuning within T4 VRAM limits without OOM failures

Usage

The adapter must be attached to the base model at full precision. Loading the 4-bit quantized base and merging the 16-bit adapter directly in VRAM introduces rounding errors and degrades transcription quality — load the base model in torch.float16 instead.

pip install -U transformers peft accelerate qwen-vl-utils "torchao>0.16.0"

import torch
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info

base_model_id = "Qwen/Qwen2-VL-2B-Instruct"
adapter_id = "Hatim2221/Mubsir-vl-arabic-htr-adapter"

# Load base model in float16 (do NOT load in 4-bit for merging/inference)
model = Qwen2VLForConditionalGeneration.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(base_model_id)

# Attach the LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "path/to/handwritten_image.jpg"},
            {"type": "text", "text": "Transcribe the Arabic handwritten text in this image."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)

print(output_text[0])

Merging the adapter yourself

If you want to produce your own merged, standalone checkpoint instead of using the pre-merged Hatim2221/Mubsir-Qwen-2B-VL:

merged_model = model.merge_and_unload()
merged_model.save_pretrained("mubsir-merged-fp16")
processor.save_pretrained("mubsir-merged-fp16")

Always merge from a float16 (or bfloat16) base — merging into a 4-bit quantized base will silently corrupt the adapter's learned weights.

Intended Use

• Digitizing Arabic handwritten documents, notes, and manuscripts
• Research and benchmarking on Arabic HTR
• As a base for further fine-tuning on domain-specific handwriting (e.g., historical manuscripts, forms, exam scripts)

Limitations

• Trained on the KHATT dataset; performance may degrade on handwriting styles, scripts, or document layouts (e.g., dense marginalia, mixed-language text) that differ significantly from the training distribution.
• As with any HTR system, accuracy depends on image quality — resolution, lighting, and skew can affect output.
• Not evaluated for languages or scripts other than Arabic.

Citation

If you use this adapter, please consider citing the base model and the KHATT dataset:

@misc{qwen2vl,
  title={Qwen2-VL},
  author={Qwen Team},
  year={2024},
  url={https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct}
}

Acknowledgements

• Base architecture: Qwen2-VL by the Qwen Team
• Training data: KHATT dataset