translategemma-4b-it-int8-ov

TranslateGemma 4B IT — OpenVINO INT8

Unofficial conversion of google/translategemma (translategemma-4b-it) to OpenVINO IR with NNCF INT8 weights.

This is not Google's original checkpoint. Weights remain under the Gemma Terms of Use. See NOTICE and LICENSE.

Files

• openvino_language_model.xml / .bin — language graph, INT8 (~3.7 GB)
• openvino_text_embeddings_model.xml / .bin — text embeddings, INT8 (~641 MB)
• openvino_vision_embeddings_model.xml / .bin — stub (not SigLIP). VLMPipeline still requires these filenames
• tokenizer / detokenizer IR plus tokenizer.json / tokenizer_config.json — same vocab, HF-format copy plus compiled OpenVINO tokenizer

Text-only generate. The original vision encoder was omitted because compiling and running full SigLIP with this 4B VLM can exceed 12 GB device memory on generate even when language weights are INT8. Restore a real vision IR into those two filenames if you need image input.

Usage

This folder is already converted. Do not re-run Optimum-Intel, NNCF, or the vision stub. Download it, load VLMPipeline, generate.

Accept the Gemma terms on this page, then:

pip install huggingface_hub openvino openvino-genai
hf download light434/translategemma-4b-it-int8-ov --local-dir translategemma-4b-it-int8-ov

import openvino_genai as ov_genai

# "GPU" = Intel GPU (Arc / iGPU). Not NVIDIA CUDA.
# Use "CPU" if OpenVINO does not list an Intel GPU (typical Kaggle NVIDIA runtimes).
pipe = ov_genai.VLMPipeline("translategemma-4b-it-int8-ov", "CPU")
pipe.set_chat_template("{{ bos_token }}{{ messages[-1]['content'] }}")

text = "The two fractions are different."
prompt = (
    "<start_of_turn>user\n"
    "You are a professional English (en) to Hindi (hi) translator. "
    "Your goal is to accurately convey the meaning and nuances of the original English text "
    "while adhering to Hindi grammar, vocabulary, and cultural sensitivities.\n"
    "Produce only the Hindi translation, without any additional explanations or commentary. "
    "Please translate the following English text into Hindi:\n\n\n"
    f"{text}"
    "<end_of_turn>\n"
    "<start_of_turn>model\n"
)

cfg = pipe.get_generation_config()
cfg.max_new_tokens = 512
cfg.max_length = 2048 + 512
cfg.do_sample = False
cfg.apply_chat_template = False
cfg.stop_token_ids = {1, 106}  # <eos>, <end_of_turn> — required or generate runs to 512 tokens
# Do not pass images=[]. Text-only; vision IR is a stub.
print(pipe.generate(prompt, generation_config=cfg))

Stop on Gemma <end_of_turn> (token 106), not only <eos> (1). Without 106, GenAI keeps decoding until max_new_tokens (~11 s) after a short translation. Cap max_length (default is 2^64-1). Do not pass images=[] (can OOM on 12 GB). These are runtime settings, not a bad INT8 checkpoint. Official TranslateGemma chat templates that enumerate every language can fail MiniJinja; the passthrough template plus the trained user-turn string is enough.

Runtime: OpenVINO GenAI (tested 2026.3). Export used Optimum-Intel / NNCF on CPU PyTorch (+cpu). No NVIDIA/CUDA.

WMT24++ subset (this INT8 IR, not Google’s 55-lang MetricX/COMET): en-es_MX, 960 segments (998 in file; 38 canary/bad skipped), greedy, official user-turn, stop_token_ids={1,106}. sacrebleu vs post-edit target: BLEU 34.83, chrF 62.1, chrF++ 60.01. No MetricX/COMET on this IR. Google’s 4B BF16 on this pair is MetricX 2.51 ↓ (appendix Table 4), not BLEU. 55-lang average: MetricX 5.32, COMET 81.6. Technical report.

Provenance

• Base: Google TranslateGemma 4B IT (Gemma 3 family). Google fine-tuned Gemma 3 → TranslateGemma. This repo did not fine-tune. It is an NNCF INT8 OpenVINO export of that checkpoint.
• Conversion: Optimum-Intel image-text-to-text export, then NNCF INT8 (language + text embeddings)
• Vision graph in this folder is a zeros stub matching output shape [1, 256, 2560]
• Modified files: the OpenVINO .xml / .bin subgraphs

License

Gemma Terms of Use: https://ai.google.dev/gemma/terms
Prohibited uses: https://ai.google.dev/gemma/prohibited_use_policy
Not affiliated with or endorsed by Google.