translategemma-4b-it-onnx-int4-dml

TranslateGemma 4B — ONNX Runtime GenAI, int4, DirectML

ONNX Runtime GenAI build (int4 weights, fp16 activations, DirectML execution provider) of the text component of google/translategemma-4b-it, Google's translation-specialized Gemma 3 model covering 55 languages.

Built for local, fully offline translation in Celerando (Menterium IT Solutions GmbH); usable with any onnxruntime-genai client on DirectX-12-capable GPUs, within the 1024-token limit described below. A CPU variant is published separately as translategemma-4b-it-onnx-int4-cpu.

Known limitation on DirectML

On DirectML, keep prompt + generated tokens ≤ 1024.

The DirectML kernel of GroupQueryAttention in ONNX Runtime does not apply the local_window_size attribute (checked in v1.24.4, v1.31.0 and main). Gemma 3 uses a 1024-token sliding window in its local attention layers, 29 of the 34 layers of this model. Beyond position 1024 these layers attend to tokens outside their window, and the model falls into repetition loops (for example 20,20,20,… on tables of figures) or ends a translation early and silently drops lines. In an isolated operator test with 1026 tokens and a window of 1024, the CPU kernel ignores a value outside the window (output 0.0), the DirectML kernel does not (output 0.998).

What a client should do on DirectML:

• set max_length to at most 1024,
• reject prompts of 1024 tokens or more,
• discard an output that reaches the limit without an end-of-turn token,
• translate long texts in sections, so that prompt and translation stay within the limit (figures and tables need far more tokens per character than running text).

The CPU execution provider applies the window correctly; the limit does not exist there (translategemma-4b-it-onnx-int4-cpu is not affected).

Conversion notes

• Text decoder extracted from the multimodal checkpoint (Gemma3ForCausalLM, model type gemma3_text); the vision tower is not included.
• Converted with the onnxruntime-genai model builder (python -m onnxruntime_genai.models.builder -p int4 -e dml).
• genai_config.json ships with eos_token_id: [1, 106] so that <end_of_turn> terminates generation (the stock config only stopped on <eos>).

Prompt format

Gemma requires the <bos> token at the start of the prompt — without it the output degenerates. The tokenizer of onnxruntime-genai does not add it automatically:

<bos><start_of_turn>user
Translate the following text into English. Reply with ONLY the translation.

Text:
Wie heißt die Hauptstadt von Frankreich?<end_of_turn>
<start_of_turn>model

License

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms (see also NOTICE; a copy of the Gemma Terms of Use and the Gemma Prohibited Use Policy is included as GEMMA_TERMS_OF_USE.txt). Use of this model is also subject to the Gemma Prohibited Use Policy. This repository redistributes a quantized derivative of TranslateGemma; all rights in the underlying model remain with Google.