opus-mt-mobile-onnx

Opus-MT for phones — sixteen directions, INT8 ONNX

Sixteen Opus-MT translation directions prepared to run offline on a phone: three quantized ONNX graphs and one binary tokenizer per direction.

Nothing here is a new model. These are conversions of Helsinki-NLP Opus-MT models, published so that a mobile app can download a language pair once instead of shipping a converter, and so the conversion is reproducible by anyone.

Directions

Directions, not languages: Opus-MT translates one way with one model, so ru-uk and uk-ru are two different sets of files. Ukrainian is covered by direct pairs with Russian rather than through English — on that pair the quality cost of two passes is particularly audible. French with Russian and Ukrainian is direct for the same reason.

Sizes run from 209 MB (ru-uk) to 264 MB (it-en). The spread comes from vocabulary, not architecture: every direction has six layers and d_model 512, but the shared vocabulary of it-en is 80 379 against 58 101 for de-en, and the difference lands in the embedding matrix.

Files in each directory

opus-mt-<pair>/
  encoder_model.onnx              INT8
  decoder_model.onnx              INT8
  decoder_with_past_model.onnx    INT8
  tokenizer.bin                   binary, see below
  manifest.json                   sizes, SHA-256, architecture, source model

Why three graphs and not one

decoder_model_merged does not quantize: the whole body of the graph hides inside a single If node, and the ONNX Runtime quantizer does not descend into subgraphs.

Dropping decoder_model and keeping only the cached decoder does not work either. It accepts an empty cache and returns logits of the right shape — but past_key_values.N.encoder.* holds projected encoder states, not history, and they are the decoder's only link to the input text. On an empty cache the model produces fluent, entirely invented translations.

Why the tokenizer is a binary file

Marian models have no tokenizer.json, and source.spm is a SentencePiece protobuf whose nmt_nfkc normalizer is a compiled 237 KB binary trie. Parsing that format on the device is a separate project with a high cost of error.

The conversion enumerates the normalizer's behaviour over code points once — about 4 800 rules — and stores it as a plain table alongside the pieces and the vocabulary. Every direction was verified piece-for-piece against the reference AutoTokenizer on a 15-sentence corpus in the source language of that direction.

Using them

The reference consumer is a Kotlin runtime (tokenizer, beam search, three ONNX sessions) built for My Chat and shared with a keyboard over an IPC contract; measured on a Galaxy Z Fold 6: median 109 ms per sentence, worst case 191 ms, after a one-off 587 ms to open a pair.

Any decoder works, with one requirement that is easy to miss:

The padding token must be forbidden during generation. Marian uses it as the decoder start token, and generation_config.json sets bad_words_ids: [[pad]]. en-uk selects it at the very first step: without the ban, generation repeats padding to the length limit and returns an empty string on a perfectly healthy model. Eleven of the twelve directions behave without the ban, which is exactly what makes the omission hard to notice.

Greedy decoding of the quantized graphs matches the fp32 reference sentence for sentence on five of the twelve directions and differs by a word or two on the rest — ordinary quantization drift, visible mostly in numerals and greetings. Beam search narrows it further.

Licence and attribution

The original models are © Helsinki-NLP, released under CC BY 4.0, and this derivative keeps that licence. If you use these files, credit the Helsinki-NLP Opus-MT project — the translation quality is theirs; only the packaging is ours.

• Models: https://huggingface.co/Helsinki-NLP
• Opus-MT / Tatoeba Challenge: Tiedemann, J. (2020), The Tatoeba Translation Challenge — Realistic Data Sets for Low Resource and Multilingual MT.

Reproducing

python3 tools/opus-mt/convert.py en uk --out dist/opus-mt
python3 tools/opus-mt/verify.py dist/opus-mt/en-uk

verify.py checks two things and refuses to skip either: that our tokenizer agrees with the reference piece for piece, and that the assembled graphs actually translate a sentence. The second check exists because the first one passed on en-uk while it was returning nothing at all.

<!-- direct-pairs:start -->

Direct directions (October 2026)

100 more directions, so that German, Italian, Portuguese, Polish and the other EU languages translate to and from Russian and Ukrainian directly instead of through English. Each source model is pinned to the Hugging Face commit shown below, so a rebuild produces the same bytes. All of them were checked the same way as the original sixteen: tokenizer piece-for-piece against AutoTokenizer on a 15-sentence corpus in the source language, plus one end-to-end translation.

Quantization was checked against fp32, not assumed. Every direction was translated over its whole corpus by the INT8 graphs and by the fp32 reference, comparing numbers, repetition loops and exact matches. Symmetric int8 weights broke some models (en-hu looped, others turned 23 into 203); asymmetric uint8 weights of the same size fixed most of them. Each direction ships the better of the two — see the Weights column. Four directions (ru-fi, ru-lt, uk-sl, uk-sv) were left out: their source models mistranslate numbers and names even in fp32, so translation goes through English.

Multilingual Tatoeba models (tc-big-*-zle, zlw-zle, en-zls, …) need a target-language token as the first input token. It is stored in tokenizer.bin format version 2: the same layout plus one int32 after decoderStart — the token id that goes first. Version 1 files (all directions without a token) are unchanged. Directions of one multilingual model share byte-identical ONNX graphs.

tc-big directions are larger (≈ 560 MB) and, measured on a Galaxy A16 (4 GB), still faster than two passes through English: median 880 ms vs 1232 ms per sentence for de-ru, with ≈ 150 MB more memory.

Licenses and attribution. Source models by Helsinki-NLP (Jörg Tiedemann et al., OPUS-MT / Tatoeba-Challenge), released under CC BY 4.0 or Apache-2.0 as listed per direction. These files are format conversions (ONNX export, INT8 quantization, binary tokenizer) and keep the license of their source model.

<!-- direct-pairs:end -->