alim-arabic-models

ALIM Arabic Models — diacritization (prose + verse) + meter classification

Character-level neural heads from the ALIM neuro-symbolic Arabic project: two specialist diacritizers (12M params each, BiLSTM + multi-head attention, three heads: vowel / shadda / tanwin) and one 16-class Arabic meter classifier (char-embedding + 2-layer BiLSTM). Weights are safetensors; every number below is measured on fixed eval sets with seed 42 — see Evaluation.

alim-arabic-models/
├── README.md                  # this file
├── inference_example.py       # self-contained CPU inference (torch + safetensors)
├── diac-prose/                # prose diacritizer  — prose DER 12.9%
│   ├── model.safetensors, config.json, README.md
├── diac-verse/                # verse diacritizer  — verse DER 11.7%
│   ├── model.safetensors, config.json, README.md
└── meter-16/                  # 16 canonical buhur — test 79.6%, gold-327 84.4%
    ├── model.safetensors, config.json, vocab.json, idx_to_meter.json, README.md

Why two diacritizers

A single 12M model cannot serve both domains (measured share curve, same 10-epoch budget, seed 42):

Each domain without its data collapses to ~46%. Route by source: poetry corpora → diac-verse, everything else → diac-prose. Per-class recall (prose / verse): bare 93/95%, fatha 88/89%, damma 88/81%, kasra 90/89%, sukun 90/91%, tanwin 83/86%, shadda 94/95%.

Quick start

# pip install torch safetensors
import sys
sys.path.insert(0, ".")
from inference_example import vocalize, predict_meter
print(vocalize("diac-prose", "العلم نور يهدي صاحبه الى الخير"))
print(predict_meter("meter-16", "قفا نبك من ذكرى حبيب ومنزل", "بسقط اللوى بين الدخول فحومل"))

Tokenizer (both diacritizers): ord(char) % 50000 per Arabic letter; diacritics and spaces carry no ids; pad id 0. Meter input: "sadr | ajuz", first 128 chars, char ids from meter-16/vocab.json.

Training recipes (reproducible)

• diac-prose: 10,802 synthetic words + 205k real fully-vocalized prose lines (50%+ density, eval-deduped), 10 epochs, AdamW 1e-3, batch 32, class weights + nunation augmentation, seed 42. Loss 1.03 → 0.48.
• diac-verse: same recipe on 54k vocalized poetry shatrs (ashaar).
• meter-16: 36,768 bayts / 16 canonical families, class weights clamped 10× (raw 766× inverse-frequency collapses training below chance — documented failure), label smoothing 0.1, 10 epochs. Test 79.6% (was 68.8% unweighted); tail classes (n≤10) score above zero.

Evaluation protocol

• Prose: Shamela clean-2k (2,000 fully-vocalized lines, 0 eval contamination — train lines exact-match deduped against it), DER over all letters + marked recall + per-vowel/shadda/tanwin recall.
• Verse: 313 bayts / 15 families, density ≥ 0.8, row meter labels mapped to canonical families; DER argmax. (A 32-bayt easy-only subset twice gave false readings — retired.)
• Meter: held-out test (2,177) + hand gold-327 at bahr-family level. IAA flag: the 327-set is 25 gold + 302 silver-candidate rows with no second-annotator IAA — 84.4% is diagnostic until IAA lands (plan §5.3).
• Tiny unseeded runs show ±30pt noise: all runs use seed 42.

Limitations

• Verse gold is partially vocalized and noisy: ~8pp of verse DER is a partial-gold artifact; Arkashaar spellings vary (hamza seats).
• Meter-constrained decoding (family templates) hurts variant verses (مجزوء etc.) — ship argmax for verse until sub-meter-aware constraints land.
• Prose SOTA reference is 6.6% DER; these 12M specialists sit at ~12%.

Version / provenance

• v1.0 (2026-09-08): diac-prose ← diac_mix100k · diac-verse ← diac_poetryonly · meter-16 ← meter16w. Source: ALIM project (neuro-symbolic Arabic: sarf/nahw/prosody rules + neural heads).
• License: MIT. Contact via repo discussions.