sloth-ime-models

Sloth IME models — Zhuyin → Traditional Chinese, fully on-device

The model family behind Slothing (樹懶智慧輸入法), all trained from scratch:

The sections below document the 25M reference encoder in depth; the 12M shipping encoder and the 60M decoder are covered in the latency/files sections.

60M next-word decoder (v2.1)

Dense-Qwen3.5 (Gated DeltaNet + full attention every 4th layer), 16k word-piece vocab (next word = one forward, 8.5 ms/word on BOOX @4t). Trained on 6.1M zh-TW sentences, then register-fine-tuned on 149k PTT/Dcard chat sentences (--init-from, 2ep):

Honesty note. Earlier versions reported 47.3/75.8 — that held-out was saturated with near-duplicates of the (small) training set and rewarded memorization; on genuinely fresh text that model scored 4.6/12.0. All predictor numbers here use never-seen fresh corpora. Reproduce: predictor_qwen35.py + build_corpus_big.py (in the GitHub repo) — see the Reproducibility section of the project README.

SlothE-T — Ternary Zhuyin → Traditional Chinese conversion encoder

The conversion model behind Slothing (懶音輸入法): a libchewing-free, on-device LLM-powered 注音 (Bopomofo) input method that does 免選字 (candidate-free) whole-sentence conversion — you type the phonetic keystream, it emits the sentence, with no candidate list to scroll.

• 25M parameters, ternary weights (W1.58A8), bidirectional encoder.
• ~7 MB ternary-packed GGUF (TQ2_0) · 99 MB fp32 master.
• Runs on-device across four frontends: fcitx5, IBus, Android, web.

What it does

Given a sequence of Zhuyin syllables (the input keystream, e.g. ㄋㄧˇ ㄏㄠˇ), the model emits the Traditional Chinese characters for each position as an aligned sequence-labeling task. The output head is phonetic-legality-masked: at each position only the ~1–50 characters that are legal readings of that syllable are scored, out of the full 8342-char vocabulary. This is what replaces the traditional IME 選字 (candidate-selection) step.

Architecture

Evaluation — honest held-out

Measured on 500 fresh zh-TW sentences (C4-zh-TW, offset far past the training window, filtered to be absent from the training corpus), then g2pW-labeled.

Note on 免選字. Earlier project numbers (~84 %) were inflated by a benchmark leak — the reference set had been sampled from the training corpus, rewarding memorization. 76 % is the honest whole-sentence held-out figure. homophone-hard and toneless are leak-clean throughout. On honest data this 25M ternary model beats the previously-shipped 12M int8 on 免選字 and homophone-hard while being smaller and faster.

Quality vs. latency

Latency is per-6-syllable decode on a BOOX (Snapdragon 662, no-dotprod ARM). 4M and 12M are measured (ORT int8). Update (2026-07-17): the 25M ternary's ~9 ms was a projection that real-device measurement later corrected — measured on the BOOX (ggml/libslothe, single 6-syllable forward): 18.5 ms at 4 threads / 31.9 ms at 2 threads (the projection missed that the per-kernel TQ2_0 speedup does not compound at full-model scale on a no-dotprod core, and an early 8-thread default dragged the A53 little cores — fixed). The Pareto story survives in a different form: a 12M ternary sibling (dim 256×12 layers, zero TQ2_0 padding tax, in this repo) measures 9.3 ms @4t / 15.8 ms @2t on the same device at 84 % 免選字 / 84 % homophone — the honest heir to the "~9 ms" claim.

_{* measured on-device (BOOX SD662, ggml/TQ2_0, 4 threads; the 25M's earlier "~9 ms" was a projection — see the note above). † 230-sentence reference set (the 25M measures 85.7 % on that same set on-device); the 免選字 columns above it use the 500-sentence held-out set.}

Training recipe (teacher-free)

• Direct cross-entropy on a g2pW-labeled zh-TW corpus, with label smoothing 0.1.
• Long OneCycleLR schedule (32 epochs), early-stopped at the peak (epoch 24) — the model overfits after (73 % 免選字 by ep32).
• 8-adjacency keyboard-error noise (TAAI-2024 error model): simulated mis-keys are constrained to the QWERTY physical 8-neighbourhood rather than any edit-distance-1 syllable.
• DDP on 2× RTX 5090.

Knowledge distillation from a 32M teacher was tried and only matched (never beat) this teacher-free recipe, so the teacher was dropped — simpler and ~2.4× faster to train. The two levers that carry the quality are (a) label smoothing (a teacher-free regularizer; +5 免選字 over plain CE) and (b) enough epochs to exploit the no-overfit headroom.

Deployment / inference

The ternary weights are shipped as a GGUF using ggml's mainline TQ2_0 ternary type, so the model runs on stock llama.cpp CPU kernels (ARM NEON, x86 AVX2, and the generic/WASM path). Measured vec_dot throughput on x86 AVX2: TQ2_0 ≈ 2.3× int8 (Q8_0) — the fastest quant type in the table — and ~2× on a no-dotprod ARM (Snapdragon 662). The ternary win is partly memory-bandwidth (2 bits/weight = 4× fewer weight bytes), so it holds on both weak and accelerator-equipped CPUs.

256-alignment note. TQ2_0 uses 256-element blocks, so the model's in-features (352, 960) are zero-padded up to the next multiple of 256 in the GGUF (352 → 512, 960 → 1024). The padding is exact ternary-zero (loss-free) but costs some MACs; a future model with 256-aligned dims removes the tax.

Files

Intended use & limits

• Intended: on-device Traditional-Chinese Bopomofo input (Taiwan readings).
• Out of scope: Simplified Chinese, mainland pinyin readings, free-form generation. The model only scores per-position legal characters — it is a converter, not a chat model.
• The whole-sentence (免選字) metric is honest-held-out and modest (76 %) by design: it is measured on unseen sentences, not the training distribution.

Citation / related work

The keyboard-error model and the 免選字-vs-選字 framing are discussed in the project's docs/RELATED-WORK.md, which positions this work against the TAAI-2024 cross-multi-IME system (李偉安, 葉展維, 張嘉惠, National Central University).