opus-mt-gl-pt-onnx

opus-mt-gl-pt-onnx

ONNX export of Helsinki-NLP/opus-mt-gl-pt, for use with optimum.onnxruntime.ORTModelForSeq2SeqLM.

Licence

Apache-2.0, taken from the base model card (Helsinki-NLP/opus-mt-gl-pt).

Export command

optimum-cli export onnx --model Helsinki-NLP/opus-mt-gl-pt --task text2text-generation-with-past --no-post-process gl-pt

int8 dynamic quantization was then applied to each ONNX file with onnxruntime.quantization.quantize_dynamic (QUInt8).

File layout

encoder_model.onnx
decoder_model.onnx
decoder_with_past_model.onnx
config.json / generation_config.json
source.spm / target.spm / vocab.json / tokenizer_config.json / special_tokens_map.json
int8/
  encoder_model.onnx
  decoder_model.onnx
  decoder_with_past_model.onnx

Sanity check

3 short sentences, PyTorch original vs this ONNX export, num_beams=4, max_new_tokens=48:

fp32 matches PyTorch on: 3/3. int8 output recorded above (not gated).

Usage

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM

model_id = "TigreGotico/opus-mt-gl-pt-onnx"
tok = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSeq2SeqLM.from_pretrained(model_id)

inputs = tok("Ola, como estas?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=48)
print(tok.decode(out[0], skip_special_tokens=True))

For the int8 quantized version, load from the int8/ subfolder:

model = ORTModelForSeq2SeqLM.from_pretrained(model_id, subfolder="int8")