Tashkeel-350M-v2-LiteRT

Tashkeel-350M-v2 — LiteRT-LM (on-device Arabic diacritization)

On-device conversion of Etherll/Tashkeel-350M-v2 — an Arabic diacritization (tashkeel) fine-tune of ibm-granite/granite-4.0-h-350m trained on Misraj/Sadeed_Tashkeela — to .litertlm bundles for the LiteRT-LM runtime. All credit for the model itself goes to its author; this repo only packages it for phones and desktops.

Requires litert-lm ≥ 0.15 to run (the Mamba2 hybrid state binds through the ExecutorMetadata section). CPU is the backend to use on phones. The int8 file does run on mobile GPUs — on a Galaxy S26 (Adreno) and a Pixel 8a (Mali) it delegates fully to LiteRT GPU and diacritizes the probe correctly (litert-lm 0.16.0, 2026-09-05) — but on both phones the CPU decodes faster (S26: 69 vs 46 tok/s; Pixel 8a: 48 vs 18) at a third to a half of the GPU's peak memory and without its 22–25 s engine init. An earlier version of this card said the selective-scan ops exceed the GPU delegate's rank limit; that describes the base model's CPU-lineage exports and did not hold for this bundle when it was gated. The fp16 file was measured on the CPU only.

Which file: fp16 is the exact-parity build — greedy output byte-identical to the HF fp32 reference on all 10 diacritization probes. int8 is 38% of the size; its two misses are single-diacritic greedy flips (both probes are byte-exact on the float parent, i.e. quantization cost, not conversion error — the same 350M-scale int8 sensitivity the base model's card documents). On phones the CPU runtime unpacks fp16 to fp32 in RAM, so int8 is the practical mobile choice; use fp16 where RAM allows.

Conversion & verification

Converted with hf-to-litertlm (family recipe granite_work/convert_granite4h.py, 2026-08-25):

• The checkpoint's chat template (byte-equal to the granite base's, 6418/6418) is embedded and applied at runtime.
• The metadata start token is dropped — granite's template has no leading BOS, and at 350M scale a prepended <|end_of_text|> flips correct diacritization into garbage (measured on this checkpoint).
• Task gate: the model card's worked example plus nine undiacritized MSA probes, bundle vs HF fp32 greedy on identical rendered strings (granite_work/gate_tashkeel.py).
• One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (my open report). In a probe that asks three questions (What is the capital of France? / What is 17 times 23? / What color is the sky on a clear day?) first in a pirate-persona conversation and then in a conversation told to answer in one word, on the same Engine against a fresh Engine in a new process (answers capped at 200 tokens; CPU backend; litert-lm 0.18.0 on a Mac), the _int8 file answered the capital question with an empty reply on the same Engine against "The answer is Paris." on a fresh one, in both of its runs (a probe is run a second time when its first run shows a difference or when the fresh Engine misses one of the three expected answers: Paris, 391, blue). The _fp16 file gave all three expected answers on both Engines in its one run. It answered the capital question with a sentence ("The capital of France is Paris.") on the same Engine against the one word "Paris" on a fresh one. The probe's questions are outside this model's task (Arabic diacritization), so the probe shows the Engine's behaviour, not the model's quality. Those three short questions say nothing about longer conversations. Create a new Engine for each conversation. The first conversation on an Engine is unaffected.

Performance (Apple M4 Max, litert-lm 0.16.0, -p 256 -d 256 --runs 3 --cache no, CPU)

Performance (Android, litert-lm 0.16.0 litert_lm_advanced_main, 205-token prompt, --benchmark, 2 cold runs per row)

Each GPU/CPU pair is the same handset, binary and prompt, taken 2026-09-05 with the phone cooled between runs; the gate prompt was the trained form above and both backends returned the correct diacritization. The same rows, with conditions, are in this repo's litertlm_manifest.json.

Usage

The model diacritizes the text you send as a user message in its trained prompt form:

pip install litert-lm
litert-lm run Tashkeel-350M-v2_fp16.litertlm --backend cpu \
  --prompt 'قم بتشكيل هذا النص :
السلام عليكم'

Output: السَّلَامُ عَلَيْكُمْ

License

apache-2.0, inherited from the source model and its granite base.