Downloads · 30 days
79
5% of all-time downloads
HarmenWessels/gemma-4-12B-it-qat-int4-ov
gemma-4-12B-it-qat-int4-ov is a image-text-to-text model from HarmenWessels. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. The card lists the license as gemma.
OpenVINO IR of Google's gemma-4-12B-it QAT checkpoint, weight-compressed to INT4 symmetric, group size 32 (Q40-grid-matched, so int4 ≈ bf16 by construction). Converted with optimum-intel PR 1770
Downloads · 30 days
79
5% of all-time downloads
All-time downloads
1.6K
Public
Repo size
8.8 GB
Likes
3
Public
Click a slice to open those files.
.bin8.2 GB · 100%
From the Hugging Face model README
OpenVINO IR of Google's gemma-4-12B-it QAT checkpoint,
weight-compressed to INT4 symmetric, group size 32 (Q4_0-grid-matched, so int4 ≈ bf16 by
construction). Converted with optimum-intel PR #1770
(gemma4_unified export support).
The stock export bakes rt_info ACTIVATIONS_SCALE_FACTOR = 8.0 for every language-model
submodel (convert.py:118).
That value is fine for the smaller Gemma 4 variants, but too small for the 12B's larger
activations (hidden 3840, 48 layers) — they overflow FP16's ~65504 range, so f16/bf16 inference
produces garbage and you're forced to slow f32. This artifact bakes ACTIVATIONS_SCALE_FACTOR = 64.0,
which keeps activations in range and lets the model run coherently at f16.
ACTIVATIONS_SCALE_FACTOR | f16 on Intel iGPU |
|---|---|
| 8.0 (stock export) | garbage |
| 64.0 (this build) | coherent ✓ |
Requires optimum-intel with PR #1770 (until merged) and transformers 5.10:
from optimum.intel import OVModelForVisualCausalLM
from transformers import AutoTokenizer
model = OVModelForVisualCausalLM.from_pretrained(
"HarmenWessels/gemma-4-12B-it-qat-int4-ov", device="GPU") # f16 by default
tok = AutoTokenizer.from_pretrained("HarmenWessels/gemma-4-12B-it-qat-int4-ov")
msgs = [{"role": "user", "content": [{"type": "text", "text": "Write a Python function that reverses a string."}]}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt")
print(tok.decode(model.generate(**inputs, max_new_tokens=256)[0]))
OpenVINO GenAI's VLMPipeline does not yet dispatch gemma4_unified — use the optimum path above.
optimum-cli export openvino -m google/gemma-4-12B-it-qat-q4_0-unquantized \
--task image-text-to-text --weight-format int4 --sym --group-size 32 --ratio 1.0 \
gemma-4-12B-it-qat-int4-ov
# then edit openvino_language_model.xml rt_info: ACTIVATIONS_SCALE_FACTOR 8.0 -> 64.0
transformers==5.10, optimum-intel PR #1770. Self-contained (model + tokenizer/detokenizer IRs).
Gemma Terms of Use — inherited from the base model.
This model is measured in core-ultra-llm-server — per-task pass/fail (code generation, editing, agentic tool use, analysis, autocomplete) on an Intel Core Ultra 155H iGPU, re-run whenever the engine or server changes. The leaderboard there is authoritative; this card carries no numbers so it cannot go stale. Its benchmark card (decoding, think budget) is cards/HarmenWessels__gemma-4-12B-it-qat-int4-ov.yaml.