MiLMMT-46-4B-v1.0-5bit-MLX

MiLMMT-46-4B v1.0 — 5-bit (MLX, text-only)

A 5-bit, group-size-64 MLX quantization of Xiaomi's MiLMMT-46-4B-v1.0, for on-device translation on Apple Silicon. 46 languages.

The base checkpoint is a Gemma3ForConditionalGeneration with a SigLIP vision tower. This conversion is text-only: vision_tower and multi_modal_projector are dropped, leaving the language model. That is what mlx_lm's gemma3 loader does automatically, and it takes the repository from 8.6 GB (bf16, with vision) to 2.5 GB.

Measured against full precision

The whole ladder was benched against the unquantized bf16 model on FLORES+ devtest, en→X, 45 languages × 100 sentences (4,500 segments per rung), chrF++ against the gold human references. Significance is sacrebleu's paired bootstrap (2,000 resamples) over the pooled 4,500 sentences.

5-bit is statistically indistinguishable from full precision — Δ −0.04, 95% CI ±0.43, and a 23/45 language split that is a coin flip. 4-bit costs a real ~0.9 chrF++, significant at p=0.0005 with a 40/45 split, concentrated in the lower-resource tail (Turkish −1.98, Uzbek −1.36, Tamil −1.33).

A blind Codex judge (different model family, scoring 450 items against the same gold references) independently put 4-bit 1.00 points below bf16 on a 0–100 scale — agreeing with chrF++ to within 0.12. A per-sentence scan of all 4,500 outputs per rung found no empty outputs, truncations, or repetition loops at any precision.

Earlier reference-free pairwise judging of 4-bit against 5-bit and 8-bit found no difference. That was a limitation of the method, not a property of the model: comparing two quantized rungs to each other cannot locate a floor, and without a gold reference the judges could not resolve a sub-point gap.

This is the recommended build. It is the lossless floor: the smallest rung that is statistically indistinguishable from the unquantized model. 6-bit and 8-bit cost more storage for no measurable gain; 4-bit is measurably behind.

Usage

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("translate-studio/MiLMMT-46-4B-v1.0-5bit-MLX")

src, tgt = "Chinese (Simplified)", "English"
text = "我爱机器翻译"
prompt = f"Translate this from {src} to {tgt}:\n{src}: {text}\n{tgt}:"

print(generate(model, tokenizer, prompt=prompt, max_tokens=512,
               sampler=make_sampler(temp=0.0), verbose=False))

Prompt format

Use the base model's plain block, with no BOS token and no chat template — the checkpoint's chat_template.jinja simply concatenates message contents, so there are no turn markers:

Translate this from <source language name> to <target language name>:
<source language name>: <source sentence>
<target language name>:

Language names must be written as the base model card lists them (e.g. Chinese (Simplified), Chinese (Traditional), English).

Note on stop tokens

config.json carries only eos_token_id: 1, while generation_config.json lists [1, 106] (106 is <end_of_turn>). Loaders that read the stop id from config.json alone can miss 106 and run on to max_tokens. Honour both.

Supported languages

Arabic, Azerbaijani, Bulgarian, Bengali, Catalan, Czech, Danish, German, Greek, English, Spanish, Persian, Finnish, French, Hebrew, Hindi, Croatian, Hungarian, Indonesian, Italian, Japanese, Kazakh, Khmer, Korean, Lao, Malay, Burmese, Norwegian, Dutch, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Swedish, Tamil, Thai, Tagalog, Turkish, Urdu, Uzbek, Vietnamese, Cantonese, Chinese (Simplified), Chinese (Traditional).

Quality is not guaranteed outside these 46. In particular, spot checks of this quantization found that prompts targeting Telugu, Marathi and Gujarati can return Hindi/Devanagari or mixed-script output rather than the requested language.

License & attribution

Derived from Gemma and distributed under the Gemma Terms of Use.

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.

Credit: Xiaomi Inc. (MiLMMT-46) and Google (Gemma 3 base architecture). This repository only provides an MLX-quantized, text-only copy of the released weights, redistributed under the same terms. Use restrictions in the Gemma Terms apply, including the prohibited-use policy.

Citation for the base model:

@misc{han2026referencefreeposttrainingopenlarge,
      title={Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation},
      author={Chris Han and Pengzhi Gao and Pei Fu and Jian Luan},
      year={2026},
      eprint={2608.10812},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.10812}
}