slothlm-e-12m-zhuyin

SlothLM-E 11.6M — Zhuyin (Bopomofo) → Traditional Chinese

The smallest useful Slothing model: a 3.9M-parameter bidirectional encoder found by neural architecture search (18-config Hyperband) over the sub-5M space, trained on g2pW context-aware readings (neural Taiwan polyphone disambiguation, not rule-based g2p).

Non-autoregressive: N bopomofo syllables in → N Traditional-Chinese characters out, one forward pass, constrained to each syllable's phonetically legal characters (phonetic_table.tsv, Taiwan-standard readings).

Architecture

dim 256 · depth 12 · 8 heads (KV 2) · FFN 768 · RoPE · GQA · QK-norm · RMSNorm · SwiGLU · post-embedding RMSNorm (à la ModernBERT — it won the NAS final) · char-hint conditioning channel (weight-tied to the output head, ~0 params): a third input (hints, id 0 = none, else char_id+1) trained with 0–30% of true chars revealed — feed a user's correction back and the whole sentence re-scores around it (新注音-style interactive re-scoring). The same channel carries document context: committed text before the cursor rides <pad>-id prefix positions (e.g. context 我妹妹說 flips 他很漂亮→她很漂亮). Trained with typo-noise (8% ED1-valid syllable corruption) for robust typo-repair scoring. Zero hints = normal decode.

Accuracy (sentence-exact)

*chewing floor measured on the earlier 30-case set; 230-case floor pending.

Sub-5M finding: homophone accuracy saturates near 3M params; whole-sentence

⚠️ Eval-leakage correction (2026-07-12): the original 免選字 numbers (incl. "84%"/"86%") were measured on a set SAMPLED FROM THE TRAINING CORPUS (174/174 sampled lines verbatim in training) → memorization-inflated by ~12-18pts. On a held-out set (fresh c4, excluded against training) the shipped 11.6M scores 72% 免選字 (homophone-hard 82% / toneless 79% were already clean). This 11.6M is the shipped model (int8, 13MB).

Repo renamed from slothlm-e-4m-zhuyin (legacy 3.8M) to reflect the 11.6M shipped model; the old slug redirects.

Reproducibility

Everything needed to reproduce this model from public sources is in this repo: REPRODUCE.md (step-by-step commands), reproduce/ (corpus builder, g2pW labeling pipeline, tokenizer/table builders, NAS driver, trainer, gate, ONNX export), eval/ (both test sets + the 免選字 set builder), and tokenizer/. Training data: 1.1M short TC sentences streamed from erhwenkuo/c4-chinese-zhtw, relabeled with g2pW (Interspeech 2022) context-aware Taiwan readings. NAS: 18-config Hyperband, seed 1234, rungs 1.5/4/8 epochs. Living code: GitHub.

Files

• slothe.pt — PyTorch checkpoint (training/fine-tuning)
• onnx/model.onnx (+.data) — fp32 export
• onnx/model_quantized.onnx — 5.0MB per-channel int8, lossless vs fp32 (naive per-tensor int8 loses 12pts on toneless — don't use it)
• syl_vocab.json, phonetic_table.tsv — serving-side vocab + legality table
• tokenizer/ — the char tokenizer (defines output ids)
• REPRODUCE.md, reproduce/, eval/ — full reproduction pipeline + evals

Demo

Runs in-browser via onnxruntime-web: https://huggingface.co/spaces/Luigi/slothing-web