Downloads · 30 days
26
26% of all-time downloads
TigreGotico/nllb-200-distilled-600M-onnx
nllb-200-distilled-600M-onnx is a translation model from TigreGotico. Use it when you need text moved from one language to another. It is set up for optimum. The card lists the license as cc-by-nc-4.0.
ONNX export of facebook/nllb-200-distilled-600M, a 600M-parameter multilingual translation model that covers 200 languages.
Downloads · 30 days
26
26% of all-time downloads
All-time downloads
99
Public
Repo size
10.7 GB
Likes
0
Public
Click a slice to open those files.
.onnx_data7.1 GB · 77%
From the Hugging Face model README
ONNX export of facebook/nllb-200-distilled-600M,
a 600M-parameter multilingual translation model that covers 200 languages.
Runs on CPU with ONNX Runtime. No PyTorch needed at inference time.
This model is CC-BY-NC-4.0. You cannot use it commercially.
The licence is inherited from the base model. Exporting a model to ONNX does not change its licence. Read the CC-BY-NC-4.0 terms before you use this model. If you need a translation model for a commercial product, this is not it.
fp32 graphs are at the repository root. int8 graphs are in int8/.
The tokenizer and config files at the root apply to both variants.
| File | Size |
|---|---|
encoder_model.onnx | 1.54 GiB |
decoder_model.onnx + decoder_model.onnx_data | 2.71 GiB |
decoder_with_past_model.onnx + decoder_with_past_model.onnx_data | 2.62 GiB |
tokenizer.json | 30.7 MiB |
sentencepiece.bpe.model | 4.6 MiB |
config.json, generation_config.json, special_tokens_map.json, tokenizer_config.json | small |
Total: about 7.0 GiB.
The .onnx_data files hold the weights of graphs above the 2 GiB protobuf limit.
You must download them together with their .onnx file. The graph does not
load without them.
int8/)| File | Size |
|---|---|
encoder_model.onnx | 399.6 MiB |
decoder_model.onnx + decoder_model.onnx_data | 700.0 MiB |
decoder_with_past_model.onnx + decoder_with_past_model.onnx_data | 675.7 MiB |
Total: about 1.8 GiB, about 3.9x smaller than fp32.
int8/ also carries its own config.json and generation_config.json so that
subfolder="int8" loads on its own. The tokenizer still comes from the root.
optimum-cli export onnx \
--model facebook/nllb-200-distilled-600M \
--task text2text-generation-with-past \
--no-post-process \
fp32/
--no-post-process is required. The default post-processing step calls
ModelProto.ByteSize() on the decoder graph, which is above 2 GiB, and protobuf
raises EncodeError instead of returning a size. The flag also keeps the two
decoder graphs separate instead of merging them into decoder_model_merged.onnx.
The int8 variant was produced per graph with ONNX Runtime dynamic quantization:
from onnxruntime.quantization import quantize_dynamic, QuantType
quantize_dynamic(
"fp32/encoder_model.onnx",
"int8/encoder_model.onnx",
weight_type=QuantType.QInt8,
per_channel=True,
reduce_range=False,
use_external_data_format=True, # decoder graphs only
extra_options={"EnableSubgraph": True},
)
ONNX Runtime writes external data as <name>.onnx.data. Those files were renamed
to optimum's <name>.onnx_data convention, because from_pretrained does not
download the first form and the graph then fails to load.
Exported with optimum 2.1.0, transformers 4.57.6, onnx 1.17.0, onnxruntime 1.26.0, torch 2.13.0. CPU only.
15 sentences over 8 language pairs (en→pt, en→gl, en→ca, en→eu, en→ar, en→ru,
pt→en, es→en) were translated with num_beams=4, max_new_tokens=64,
do_sample=False, and the correct forced_bos_token_id. The output strings were
compared to the transformers PyTorch model, exact string match:
| Variant | Exact match vs PyTorch |
|---|---|
| fp32 | 15/15 (100%) |
| int8 | 12/15 (80%) |
The three int8 differences are all fluent alternative wordings, not corrupted output. Dynamic quantization shifts the beam scores just enough to change the winning hypothesis:
| Pair | fp32 / PyTorch | int8 |
|---|---|---|
| en→gl | Os pescadores volvieron ao porto antes de que chegara a tempestade. | Os pescadores regresaron ao porto antes de que chegara a tempestade. |
| en→eu | Umeak elizaren ondoko plazan jolasten ari dira. | Umeak jolasean ari dira elizaren ondoan. |
| en→ar | طلب من السائق أن يتوقف عند الزاوية التالية. | طلب من السائق التوقف عند الزاوية التالية. |
Use int8 when memory or disk matters more than reproducing the PyTorch output token for token. Use fp32 when you need the reference behaviour.
Maximum absolute logit difference reported by the exporter's own validation was 1.6e-4, which is ordinary fp32 rounding.
NLLB uses FLORES-200 language codes, in the form eng_Latn, por_Latn,
glg_Latn, cat_Latn, eus_Latn, arb_Arab, rus_Cyrl, zho_Hans. The code
carries both the language and the script.
Two things must be set:
tokenizer.src_lang = "eng_Latn" before you tokenize.forced_bos_token_id on generate(), obtained with
tokenizer.convert_tokens_to_ids("por_Latn").If you skip forced_bos_token_id, the model picks a target language on its own
and the output is usually wrong. The full list of the 200 codes is in the
base model card.
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM
repo = "TigreGotico/nllb-200-distilled-600M-onnx"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ORTModelForSeq2SeqLM.from_pretrained(repo, use_cache=True)
# For the int8 variant instead:
# model = ORTModelForSeq2SeqLM.from_pretrained(repo, subfolder="int8", use_cache=True)
tokenizer.src_lang = "eng_Latn"
inputs = tokenizer("The library closes at eight o'clock every evening.", return_tensors="pt")
output = model.generate(
**inputs,
forced_bos_token_id=tokenizer.convert_tokens_to_ids("por_Latn"),
num_beams=4,
max_new_tokens=64,
)
print(tokenizer.batch_decode(output, skip_special_tokens=True)[0])
# A biblioteca fecha às oito horas todas as noites.
Cite the original NLLB work:
@article{nllb2022,
title={No Language Left Behind: Scaling Human-Centered Machine Translation},
author={{NLLB Team} and Costa-juss{\`a}, Marta R. and others},
year={2022},
journal={arXiv preprint arXiv:2207.04672}
}