KhattVision-Muse-Glimmer-30B-LoRA

🖋️ KhaṭṭVision

<div align="center">

A Muse Glimmer 30B LoRA for Arabic calligraphy understanding

<img src="./IMG_7275.png" alt="KhaṭṭVision icon" width="200">

Arabic OCR · Calligraphic style recognition · Theme classification · Text-region localization

<p align="center"> <a href="https://huggingface.co/datasets/MBZUAI/DuwatBench"> <img alt="DuwatBench dataset" src="https://img.shields.io/badge/Dataset-DuwatBench-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000000"> </a> <a href="https://www.kaggle.com/code/engomarnajar/notebookf34951354e"> <img alt="KhaṭṭVision fine-tuning code" src="https://img.shields.io/badge/Fine--Tuning_Code-Kaggle-20BEFF?style=for-the-badge&logo=kaggle&logoColor=ffffff" > </a> <a href="https://huggingface.co/unsloth/Muse-Glimmer-30B-unsloth-bnb-4bit"> <img alt="Muse Glimmer base checkpoint" src="https://img.shields.io/badge/Base-Muse_Glimmer_30B-6366F1?style=for-the-badge&logo=huggingface&logoColor=ffffff"> </a> <a href="https://arxiv.org/abs/2601.19898"> <img alt="DuwatBench paper" src="https://img.shields.io/badge/Paper-arXiv_2601.19898-B31B1B?style=for-the-badge&logo=arxiv&logoColor=ffffff"> </a> </p> </div>

Model description

KhaṭṭVision is a parameter-efficient adaptation of Muse Glimmer 30B for understanding artistic Arabic calligraphy. It was fine-tuned on DuwatBench to perform four complementary tasks from a single image:

1. Transcribe all visible Arabic text.
2. Recognize one or more calligraphic styles.
3. Classify the semantic theme using a closed label set.
4. Return text regions as valid JSON with normalized bounding boxes.

This repository contains a LoRA adapter, not a merged standalone model. The compatible Muse Glimmer base checkpoint is required for inference.

[!IMPORTANT]

🔬 Research opportunity: from proof of concept to robust calligraphy understanding

KhaṭṭVision should be viewed as a research proof of concept, not as a solved Arabic calligraphy OCR system. The current results demonstrate that targeted multimodal fine-tuning can substantially improve OCR, structured output, style recognition, and text-region localization on Arabic calligraphy.

Dense and highly decorative compositions—especially those with overlapping letters, unusual reading order, heavy ornamentation, or low visual separation—remain challenging and can trigger fluent but incorrect transcriptions. This limitation is itself an important research opportunity.

The next stage is to build a dedicated complex-calligraphy training set with expert-verified transcriptions, reading-order annotations, difficult negative examples, higher-resolution crops, and explicit uncertainty supervision. KhaṭṭVision establishes the feasibility of the approach and provides a measurable baseline for the next generation of models.

Supported tasks

Full-image Arabic OCR

The model transcribes the Arabic content of the complete image and returns text only. When multiple text segments are visible, they are separated by line breaks.

Calligraphic style recognition

Style recognition uses the following six canonical labels. An image may contain more than one style.

Theme classification

Theme prediction is constrained to one of nine labels:

dedication
devotional invocation
hadith
names of Allah
names of companions
names of the Prophet
non-religious
personal/place name
quranic

Structured image understanding

Structured analysis returns valid JSON with the following schema:

{
  "styles": ["Diwani"],
  "theme": "quranic",
  "regions": [
    {
      "bbox_1000_xywh": [100, 100, 700, 300],
      "text": "النص العربي"
    }
  ]
}

bbox_1000_xywh means [x, y, width, height], normalized to an integer canvas from 0 to 1000. Coordinates can be converted to pixels as follows:

x_px = x * image_width / 1000
y_px = y * image_height / 1000
w_px = width * image_width / 1000
h_px = height * image_height / 1000

Training data

KhaṭṭVision was fine-tuned on MBZUAI/DuwatBench, a curated benchmark containing 1,272 Arabic calligraphy images annotated with text, calligraphic style, thematic category, and text-region bounding boxes.

Leakage-aware data split

A grouped split was used to reduce train/test contamination. Images were grouped using:

• Normalized full transcriptions.
• Exact source URLs.
• Perceptual similarity for near-duplicate images.

Two hundred reproducible grouped split candidates were evaluated for size and label-distribution balance. The selected split used seed 3909.

The split groups were asserted to be mutually disjoint before fine-tuning.

Training task mixture

Each source image was converted into one or more instruction-following examples:

• Structured JSON analysis.
• Full-image OCR.
• Cropped-region OCR.
• Explicit theme classification.

Style labels were supervised inside the structured examples. Assistant-response masking was used so that the loss was calculated only on target responses.

Training procedure

The base checkpoint was loaded in 4-bit precision and adapted using LoRA across both vision and language components.

LoRA was applied to attention and MLP projections in the language model and to selected vision layers. The full run completed on a Kaggle dual-T4 environment in approximately 6 hours and 35 minutes.

Final training statistics

Evaluation

Evaluation used a fixed, randomly selected subset of 50 held-out test images. The base model and fine-tuned adapter were evaluated on exactly the same rows with deterministic decoding. OCR metrics were computed after applying the same Arabic normalization procedure to references and predictions.

OCR results

CER and WER may exceed 1.0 when the prediction contains many insertions relative to a short reference.

Structured-output results

These results show clear domain adaptation, especially for OCR formatting, style recognition, JSON validity, and region localization. Theme recognition remains the weakest classification component.

<details> <summary>Fixed evaluation row indices</summary>

212, 79, 565, 832, 467, 837, 186, 239, 83, 777,
449, 475, 270, 841, 808, 93, 408, 92, 1255, 396,
1264, 282, 220, 235, 842, 835, 854, 184, 441, 662,
847, 225, 113, 1211, 63, 1225, 571, 292, 533, 84,
1259, 274, 540, 773, 848, 190, 104, 1256, 810, 706

</details>

Usage

Installation

pip install -U torch transformers peft accelerate bitsandbytes pillow

Load the adapter

import copy
import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

BASE_MODEL = "unsloth/Muse-Glimmer-30B-unsloth-bnb-4bit"

ADAPTER_MODEL = (
    "Omartificial-Intelligence-Space/"
    "KhattVision-Muse-Glimmer-30B-LoRA"
)

base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_MODEL,
    is_trainable=False,
)

model.eval()

processor = AutoProcessor.from_pretrained(ADAPTER_MODEL)

Full-image OCR

image = Image.open("calligraphy.jpg").convert("RGB")

instruction = (
    "اقرأ جميع النصوص العربية الظاهرة في لوحة الخط. "
    "أعد النص فقط دون شرح، وافصل المقاطع بسطر جديد."
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": instruction},
        ],
    }
]

prompt = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)

# Preserve the prompt format used during fine-tuning.
prompt += " to=user<|message|>"

inputs = processor(
    text=[prompt],
    images=[[image]],
    add_special_tokens=False,
    return_tensors="pt",
).to("cuda")

generation_config = copy.deepcopy(model.generation_config)
generation_config.max_length = None
generation_config.max_new_tokens = 192
generation_config.do_sample = False
generation_config.use_cache = True

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        generation_config=generation_config,
    )

new_tokens = output_ids[:, inputs["input_ids"].shape[1]:]

prediction = processor.tokenizer.batch_decode(
    new_tokens,
    skip_special_tokens=True,
)[0].strip()

print(prediction)

Structured analysis prompt

For structured prediction, use the same loading code with the following prompt:

instruction = """
حلّل لوحة الخط العربي وحدّد أنواع الخط، والتصنيف الموضوعي،
والنصوص العربية ومواقعها.

القيم المسموح بها في styles فقط:
["Thuluth", "Diwani", "Naskh", "Kufic", "Ruq'ah", "Nasta'liq"]

يجب أن تكون theme قيمة واحدة فقط من:
["dedication", "devotional invocation", "hadith", "names of Allah",
 "names of companions", "names of the Prophet", "non-religious",
 "personal/place name", "quranic"]

تعليمات إلزامية:
- استخدم أسماء التصنيفات السابقة حرفيًا.
- لا تستخدم مرادفات مثل religious أو Islamic أو Dua أو Quranic.
- استخدم bbox_1000_xywh بصيغة [x, y, width, height].
- يجب أن تكون جميع الإحداثيات أعدادًا صحيحة بين 0 و1000.
- أعد JSON صالحًا فقط.
- لا تضف شرحًا أو Markdown.

البنية المطلوبة:
{
  "styles": ["Diwani"],
  "theme": "quranic",
  "regions": [
    {
      "bbox_1000_xywh": [100, 100, 700, 300],
      "text": "النص العربي"
    }
  ]
}
""".strip()

Use max_new_tokens=512 for structured analysis.

Intended uses

KhaṭṭVision is intended for:

• Arabic calligraphy research.
• Exploratory OCR and transcription assistance.
• Digital-humanities prototyping.
• Calligraphic collection indexing.
• Educational demonstrations.
• Research on structured multimodal generation.

Limitations

• Decorative compositions: Dense, overlapping, curved, or highly stylized calligraphy remains difficult.
• Hallucination risk: When visual evidence is ambiguous, the model may generate a plausible Quranic verse, prayer, or religious phrase that is not present in the image.
• Small benchmark: Fine-tuning used only 1,272 source images, with strong imbalance across styles and themes.
• Theme recognition: Held-out theme accuracy was 50%, substantially below style recognition accuracy.
• Rare labels: Categories with few examples are less reliable than the dominant Quranic and devotional categories.
• Image resizing: Fine details may be lost when large images are resized to the training resolution policy.
• Bounding boxes: Region predictions are approximate and may merge adjacent text, miss decorative text, or include ornaments.
• Evaluation scope: Reported metrics use a fixed 50-image test subset, not the complete 180-image test split.
• Arabic normalization: Normalized OCR metrics do not capture every orthographic or diacritic distinction important to expert readers.

Responsible use

Do not treat generated text as an authoritative reading of religious, historical, archival, legal, or culturally sensitive material. A fluent Arabic prediction can still be visually unsupported.

Human verification against the source image is required, especially for Quranic verses, hadith, names, dates, and inscriptions.

The model should not be used to authenticate artworks, determine provenance, attribute authorship, or replace qualified calligraphers, historians, archivists, or religious scholars.

Licensing

This repository distributes a LoRA adapter. Use of the adapter is subject to the terms of the underlying Muse Glimmer base model.

DuwatBench metadata is released under Apache-2.0, while individual images may retain the licenses or usage terms of their original sources. Users are responsible for reviewing the base-model license and the rights associated with input and dataset images.

Citation

If you use KhaṭṭVision, please cite the model repository and the DuwatBench paper:

@software{nacar2026khattvision,
  author = {Omer Nacar},
  title = {KhaṭṭVision: A Muse Glimmer 30B LoRA for Arabic Calligraphy Understanding},
  year = {2026},
  url = {https://huggingface.co/Omartificial-Intelligence-Space/KhattVision-Muse-Glimmer-30B-LoRA}
}

See the official DuwatBench dataset page and paper for the dataset citation.

Acknowledgements

This work builds on the Muse Glimmer base model, Unsloth, Hugging Face Transformers and PEFT, and the DuwatBench dataset.

The project was developed by Omer Nacar through Omartificial Intelligence Space.