mt-hitz-eu-es-onnx

mt-hitz-eu-es-onnx

ONNX export of HiTZ/mt-hitz-eu-es, a MarianMT machine translation model for eu -> es produced by HiTZ, the Basque Center for Language Technology at the University of the Basque Country (UPV/EHU).

All credit for training and releasing the original model goes to the HiTZ team. This repo only contains an ONNX conversion (fp32 and int8) for fast CPU inference with optimum[onnxruntime], done by TigreGotico.

License

apache-2.0, as declared on the source model card.

Export command

optimum-cli export onnx --model HiTZ/mt-hitz-eu-es --task text2text-generation-with-past --no-post-process eu-es

int8 dynamic quantization was applied afterwards with onnxruntime.quantization.quantize_dynamic (QUInt8) into the int8/ subfolder.

File layout

eu-es/
  encoder_model.onnx              # fp32
  decoder_model.onnx              # fp32
  decoder_with_past_model.onnx    # fp32
  config.json, generation_config.json
  source.spm, target.spm, vocab.json, tokenizer_config.json, special_tokens_map.json
  int8/
    encoder_model.onnx
    decoder_model.onnx
    decoder_with_past_model.onnx

Parity vs. the original PyTorch model

Measured on 8 held-out eu->es sentences, beam search (num_beams=4, max_new_tokens=64), exact-match against the original PyTorch MarianMTModel output:

• fp32: 100.0% exact match
• int8: 75.0% exact match (semantically consistent but not gated on this number; dynamic quantization trades some fluency, especially on longer sentences, for ~2.5x smaller size and faster CPU inference)

Example (eu -> es):

• PyTorch: Hola, ¿qué tal estás?
• ONNX fp32: Hola, ¿qué tal estás?
• ONNX int8: Hola, ¿qué tal estás?

Usage

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("TigreGotico/mt-hitz-eu-es-onnx")

# fp32 (default, root of the repo)
model = ORTModelForSeq2SeqLM.from_pretrained("TigreGotico/mt-hitz-eu-es-onnx")

# int8 (smaller / faster, subfolder)
# download the repo and load from the local "int8" folder, e.g.:
# from huggingface_hub import snapshot_download
# path = snapshot_download("TigreGotico/mt-hitz-eu-es-onnx")
# model = ORTModelForSeq2SeqLM.from_pretrained(path + "/int8")

batch = tok(["your sentence here"], return_tensors="pt")
generated = model.generate(**batch, num_beams=4, max_new_tokens=64)
print(tok.batch_decode(generated, skip_special_tokens=True))

Attribution

• Original model: HiTZ/mt-hitz-eu-es by HiTZ Center (University of the Basque Country, UPV/EHU)
• ONNX export: TigreGotico