Emhotob-50M-Tashkil-v2

Emhotob-50M-Tashkil-v2

A 51.8M-parameter Llama model fine-tuned to add tashkil (diacritics / harakat) to undiacritized Arabic text. Fine-tuned from oddadmix/50M-2048-Emhotob on the full freococo/arabic_tashkil_dataset (~1.5M gold-standard vocalized pages of classical Arabic / Islamic texts).

This is v2 of the series — trained on ~20× more data than oddadmix/Emhotob-50M-Tashkil-v1 and substantially better at word preservation.

License note: the training data is licensed CC BY-NC-ND 4.0 (non-commercial, no-derivatives). This model is released under the same license to respect the upstream terms. Use accordingly.

Task framing

Prompt-masked supervised fine-tuning (causal LM). Loss is computed only on the diacritized output span:

شكِّل النص العربي التالي:
{undiacritized text}
### الرد:
{diacritized text}</s>      ← loss only here

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "oddadmix/Emhotob-50M-Tashkil-v2"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).cuda().eval()

text = "ذهب الطالب الى المدرسة"
prompt = f"شكِّل النص العربي التالي:\n{text}\n### الرد:\n"
ids = [tok.bos_token_id] + tok.encode(prompt, add_special_tokens=False)
out = model.generate(
    torch.tensor([ids]).cuda(),
    max_new_tokens=512, do_sample=False,
    eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id,
)
print(tok.decode(out[0][len(ids):], skip_special_tokens=True))

Training

• Data: full freococo set, restricted to the 1,012,933 pages whose diacritized target fits the model's 2048-token context (a max-chars 2500 pre-filter; ~32% of pages that would require truncation were dropped rather than corrupt the add-only training signal). A disjoint 4,000-row validation set was carved with a fixed seed.
• Config: 1 epoch, batch 16 × grad-accum 2 (effective 32), LR 3e-4 cosine, bf16, max_length 2048.
• Hardware: single RTX 5090, ~2 h, 31,655 steps.
• Final eval_loss: 0.0226.

Evaluation

300 held-out rows, greedy decoding.

In-domain (freococo clean held-out):

Zero-shot on the arbml test split (never seen during training):

vs v1: in-domain DER improves 1.88% → 1.32%, and the "add-only" word-preservation weak spot is largely fixed (skeleton-exact 89.7% → 97.3%). Even zero-shot on arbml, v2 preserves words far better than v1; its slightly higher DER there reflects domain shift, not a regression.

skeleton exact = strip all diacritics from the output and check it equals the input character-for-character — the strongest guarantee the model only added marks.

Limitations

• Small model (52M params); tuned on classical Arabic / Islamic prose (with § and -[NN]- structural markers preserved). Domain-shifted text may see higher DER.
• Context limited to 2048 tokens; pages longer than that were excluded from training and should be split at inference.
• Non-commercial, no-derivatives license inherited from the training data.