small100-onnx

SMaLL-100 · ONNX (int8)

English | 中文

ONNX export of SMaLL-100 — a shallow, distilled multilingual machine-translation model (distilled from M2M-100), covering 100 languages with direct any-to-any translation (no English pivot). This repo packages an int8-quantized, ~609 MB deployment set for on-device / offline use. There is no official SMaLL-100 ONNX on the Hub; this fills that gap.

Platforms · 跨平台

One model + one tokenizer.json + one lang_tokens.json, five runtimes. All follow the same recipe in USAGE.md; runnable/reference code in examples/.

✅ = runnable and verified (Flutter: the actual translator class from a shipped Android/iOS S2S translator app, real devices). 📝 = reference implementation, algorithm complete but not wired to a build.

tokenizer.json is a validated HuggingFace fast tokenizer (encoding matches the official SMaLL-100 tokenizer exactly); lang_tokens.json maps 100 language codes → token ids. The target-language token is prepended to the source (not forced_bos_token_id). 语言 token 加在源句前，各平台一致。

---

English

Why SMaLL-100

• Direct translation between any of 100 languages (no pivot → single hop).
• Shallow decoder (3 layers) → fast: ~4× faster than M2M-100 at similar quality; on a laptop CPU greedy decoding is ~150–250 ms/sentence.
• Small enough for mobile: 609 MB int8 vs ~850 MB for NLLB-200-distilled-600M.

Files — what actually needs what

Only 4 files are universal; everything else exists for one specific loading path. If you're integrating on Android/iOS/Flutter, that's all you need — ignore the rest.

The two .onnx files are int8-quantized but keep the standard optimum names (encoder_model.onnx / decoder_model_merged.onnx) so from_pretrained loads them without extra arguments; this repo ships the quantized weights only.

Model constants: d_model=1024, encoder_layers=12, decoder_layers=3, heads=16 (head_dim=64), vocab=128112, decoder_start=eos=2, pad=1, unk=3. The decoder ONNX I/O is the standard optimum merged-decoder signature (input_ids, encoder_hidden_states, encoder_attention_mask, past_key_values.{i}.{decoder,encoder}.{key,value}, use_cache_branch → logits, present.{i}...).

Usage (optimum / onnxruntime, Python)

pip install optimum[onnxruntime] transformers sentencepiece
python scripts/translate_demo.py

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from tokenization_small100 import SMALL100Tokenizer

tok = SMALL100Tokenizer.from_pretrained(".")
model = ORTModelForSeq2SeqLM.from_pretrained(
    ".", subfolder="onnx", use_merged=True, use_io_binding=False)

tok.tgt_lang = "en"                       # target token is prepended to the SOURCE
enc = tok("你好，请问最近的地铁站怎么走？", return_tensors="pt")
out = model.generate(**enc, num_beams=1, max_length=128)
print(tok.batch_decode(out, skip_special_tokens=True)[0])
# -> "Hello, what is the nearest metro station?"

Note (M2M vs SMaLL-100): SMaLL-100 does not use forced_bos_token_id. The target language is selected by setting tok.tgt_lang, which prepends the language token (e.g. en→128022, zh→128102, ja→128046, ko→128052) to the source input. Decode is greedy from decoder_start_token_id=2.

Supported languages (100)

Any-to-any: pass any of these ISO-639 codes as the target. Full code→token-id map in lang_tokens.json.

<details><summary><b>100 languages · click to expand</b></summary> </details>

Benchmarks (int8, laptop CPU arm64, greedy)

Quality is decent for a 330M distilled model; high-resource pairs are good, some place-name / register slips on harder pairs.

How it was quantized (the merged-decoder trick)

The merged decoder wraps its cached / non-cached paths in an ONNX If node. onnxruntime.quantization.quantize_dynamic skips subgraphs by default, leaving the decoder unquantized (~1.26 GB). The fix:

quantize_dynamic(src, dst, weight_type=QuantType.QInt8,
                 extra_options={"EnableSubgraph": True})

This quantizes the MatMuls inside the If branches → merged decoder ~322 MB, keeping the KV cache (so generation stays fast). See scripts/export.py.

Tokenizer

tokenizer.json is a validated HuggingFace fast tokenizer whose encoding matches the official SMaLL-100 tokenizer exactly (verified across zh/en/ja/ko). It loads in tokenizers (Rust), transformers.js, DJL (Android) and swift-transformers (iOS) — one tokenizer for every platform. It was rebuilt from Xenova/m2m100_418M (same 128112 vocab) by dropping 1053 merges that referenced out-of-vocab pieces (which made newer tokenizers reject the file), and by setting the post-processor to append </s> only. The language token is not baked in — prepend lang_tokens.json[tgt] in app code (see USAGE.md). sentencepiece.bpe.model + tokenization_small100.py remain for the upstream Python path.

License

SMaLL-100 is released under the MIT license (see the upstream model card). This repo redistributes the ONNX-converted weights under the same terms.

Upstream: SMaLL-100 (Mohammadshahi et al., EMNLP 2022), paper · model.

---

中文

SMaLL-100 的 ONNX 导出版——一个 浅层、蒸馏的多语言机器翻译模型（从 M2M-100 蒸馏而来），覆盖 100 种语言、任意语向 直译（无需经英语中转）。本仓库打包了 int8 量化、约 609 MB 的部署集，供端上 / 离线使用。目前 Hub 上没有官方的 SMaLL-100 ONNX，本仓库填补这一空缺。

为什么选 SMaLL-100

• 任意 100 种语言之间直译（无中转 → 单跳）。
• 浅层解码器（3 层） → 快：质量相近下比 M2M-100 快约 4×；笔记本 CPU 上贪心解码 约 150–250 ms/句。
• 体积适合移动端：int8 609 MB，相比之下 NLLB-200-distilled-600M 约 850 MB。

文件说明——各文件到底谁在用

真正跨平台通用的只有 4 个文件；其余的都只服务于某一条特定的加载路径。如果你在 接入 Android/iOS/Flutter，只需要这 4 个，其余可以完全忽略。

两个 .onnx 文件是 int8 量化的，但保留 optimum 标准命名 （encoder_model.onnx / decoder_model_merged.onnx），以便 from_pretrained 无需额外参数即可加载；本仓库只提供量化后的权重。

模型常量：d_model=1024、encoder_layers=12、decoder_layers=3、heads=16 （head_dim=64）、vocab=128112、decoder_start=eos=2、pad=1、unk=3。decoder 的 ONNX 输入输出是 optimum 标准合并解码器签名（input_ids、 encoder_hidden_states、encoder_attention_mask、 past_key_values.{i}.{decoder,encoder}.{key,value}、use_cache_branch → logits、present.{i}...）。

用法（optimum / onnxruntime，Python）

pip install optimum[onnxruntime] transformers sentencepiece
python scripts/translate_demo.py

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from tokenization_small100 import SMALL100Tokenizer

tok = SMALL100Tokenizer.from_pretrained(".")
model = ORTModelForSeq2SeqLM.from_pretrained(
    ".", subfolder="onnx", use_merged=True, use_io_binding=False)

tok.tgt_lang = "en"                       # 目标语言 token 会被加到「源句」前
enc = tok("你好，请问最近的地铁站怎么走？", return_tensors="pt")
out = model.generate(**enc, num_beams=1, max_length=128)
print(tok.batch_decode(out, skip_special_tokens=True)[0])
# -> "Hello, what is the nearest metro station?"

注意（M2M 与 SMaLL-100 的区别）：SMaLL-100 不使用 forced_bos_token_id。 目标语言通过设置 tok.tgt_lang 选择，它会把语言 token（如 en→128022、 zh→128102、ja→128046、ko→128052）加到源句前面。解码从 decoder_start_token_id=2 开始贪心生成。

支持的语言（100 种）

任意语向互译：把下列任一 ISO-639 code 作为目标语言传入。完整的 code→token id 映射见 lang_tokens.json。

<details><summary><b>100 种语言 · 点击展开</b></summary> </details>

基准（int8，笔记本 CPU arm64，贪心解码）

作为 330M 蒸馏模型质量尚可；高资源语对不错，较难的语对偶有地名 / 语气偏差。

量化方法（合并 decoder 的关键技巧）

合并 decoder 把「带缓存 / 不带缓存」两条路径包在一个 ONNX If 节点里。 onnxruntime.quantization.quantize_dynamic 默认不进子图，会导致 decoder 未被量化 （约 1.26 GB）。解决办法：

quantize_dynamic(src, dst, weight_type=QuantType.QInt8,
                 extra_options={"EnableSubgraph": True})

这会量化 If 分支内部的 MatMul → 合并 decoder 约 322 MB，且保留 KV 缓存（生成依旧 很快）。见 scripts/export.py。

分词器

tokenizer.json 是一份经过验证的 HuggingFace fast 分词器，编码结果与官方 SMaLL-100 分词器完全一致（中英日韩均已核对）。它能在 tokenizers(Rust)、 transformers.js、DJL(Android)、swift-transformers(iOS) 中加载——一份分词器通吃四端。 它基于 Xenova/m2m100_418M（同 128112 词表）重建：剔除了 1053 个引用越界 piece 的 merge（正是它们导致新版 tokenizers 拒绝加载），并把 post-processor 改为只追加 </s>。语言 token 不写死在分词器里——由应用层前缀 lang_tokens.json[tgt] （见 USAGE.md）。sentencepiece.bpe.model + tokenization_small100.py 保留作为 上游 Python 路径。

许可协议

SMaLL-100 以 MIT 协议发布（见上游模型卡）。本仓库以相同条款再分发 ONNX 转换后的 权重。

上游：SMaLL-100（Mohammadshahi 等，EMNLP 2022）， 论文 · 模型。