madlad400-3b-mt-onnx

madlad400-3b-mt-onnx

ONNX export of google/madlad400-3b-mt, a 3B-parameter T5 translation model that covers 419 languages.

The repository holds a full-precision (fp32) export at the root and an int8 dynamically quantized copy in int8/.

Licence

Apache-2.0, the same licence as the source model. Commercial use is permitted.

How to select the target language

MADLAD400 does not use forced_bos_token_id. You must prefix the input text with a <2xx> target-language token, where xx is the target language code:

"<2pt> The library opens at nine in the morning."

If you omit the prefix, or if you copy the NLLB forced_bos_token_id recipe, the model silently translates into the wrong language. The source language is detected automatically and is never specified.

Usage

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

repo = "TigreGotico/madlad400-3b-mt-onnx"
tok = AutoTokenizer.from_pretrained("google/madlad400-3b-mt")
model = ORTModelForSeq2SeqLM.from_pretrained(repo)          # fp32
# model = ORTModelForSeq2SeqLM.from_pretrained(repo, subfolder="int8")

inputs = tok("<2pt> The library opens at nine in the morning.",
             return_tensors="pt")
ids = model.generate(**inputs, num_beams=4, max_new_tokens=64)
print(tok.batch_decode(ids, skip_special_tokens=True)[0])
# A biblioteca abre às nove da manhã.

from_pretrained prints a message that it could not find decoder_model_merged.onnx. This is expected: the export skips the decoder-merge step (see below) and ONNX Runtime falls back to the separate decoder graphs.

Files

Total: 19 GB fp32, 4.7 GB int8.

External data — download the .onnx_data files

Each fp32 graph is larger than the 2 GB protobuf limit, so its weights live in a separate *.onnx_data file next to the .onnx file. The .onnx file is useless on its own — it is only a few hundred kilobytes of graph structure. from_pretrained fetches both, but if you download files by hand, or use allow_patterns=["*.onnx"], you must take the matching .onnx_data file too.

The int8 graphs are all under 2 GB and are self-contained.

Export

optimum-cli export onnx \
  --model google/madlad400-3b-mt \
  --task text2text-generation-with-past \
  --no-post-process \
  madlad400-3b-mt/fp32

--no-post-process is required. The decoder-merge post-processing step runs out of memory and trips a protobuf ByteSize() error on graphs above 2 GB.

int8 was produced with onnxruntime.quantization.quantize_dynamic (QuantType.QInt8, EnableSubgraph=True).

Exported with optimum on onnxruntime 1.28, transformers 4.57.6, torch 2.13 (CPU).

Parity

12 sentences over 6 language pairs (en→pt, en→gl, en→ca, en→eu, en→ar, pt→en), greedy-free beam search with num_beams=4, max_new_tokens=64, compared against the PyTorch original by exact string match.

The four int8 mismatches are synonym-level rewordings in the correct target language ("at the bakery" vs "from the bakery", المعلم vs المدرس), not failures. int8 is reported for information and is not a release gate.

The export itself validates to a maximum absolute logit difference of 2.0e-05 against the PyTorch reference.