Tashkeel-50M

Tashkeel-50M

A ~50M parameter Arabic diacritization (تشكيل) model fine-tuned from oddadmix/50M-2048-Emhotob on Misraj/Sadeed_Tashkeela.

It is a small causal LM intended for fast, on-device / low-cost Arabic tashkeel experiments.

Model details

Prompt format

Training and inference use this prompt:

قم بتشكيل هذة الجمله : {undiacritized_text}

The model should continue with the diacritized Arabic text.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ahmedsamirtarjama/Tashkeel-50M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained(model_id)
tok.padding_side = "left"
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)
model.eval()

text = "اللغة العربية لغة جميلة"
prompt = f"قم بتشكيل هذة الجمله : {text}\n"
inputs = tok(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tok.pad_token_id,
    )

print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training

Full fine-tune (not LoRA) with Hugging Face Trainer:

Approximate training recipe:

<source prompt> + <diacritized target> + </s>

Evaluation

Evaluated on Misraj/SadeedDiac-25 with standard Morph/Total DER & WER (missing GT diacritics skipped).

Mapping used below: Total ≈ (CE) (with case endings), Morph ≈ (w/o CE) (without case endings). Hallucinations ≈ share of examples skipped due to word-count mismatch.

*Tashkeel-50M DER/WER are computed only on examples where the prediction and reference have the same word count. Because most generations change length (truncation / repetition / insertions), they are skipped by the length-matching evaluator — hence the high hallucination rate. Treat the starred numbers as optimistic and not a full apples-to-apples comparison with systems that preserve word identity on nearly all examples.

For production tashkeel, prefer stronger constrained models or add decoding constraints that keep the undiacritized skeleton fixed.

Intended use

• Research and prototyping for Arabic diacritization
• Baseline for small / efficient tashkeel models
• Educational demos of causal-LM fine-tuning for sequence transduction

Limitations

• Small capacity (~50M); quality lags dedicated / large instruction models on hard classical Arabic
• Causal generation can truncate, repeat, or insert words; DER/WER only apply when word counts match
• Prompt is Arabic-instruction style; changing the prompt may degrade quality
• Not a general-purpose chat model

Citation

If you use this model, please also cite the base model and dataset:

@misc{tashkeel50m,
  title        = {Tashkeel-50M},
  author       = {Ahmed Samir},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ahmedsamirtarjama/Tashkeel-50M}}
}

• Base: oddadmix/50M-2048-Emhotob
• Data: Misraj/Sadeed_Tashkeela