gemma-4-E4B-it-int8-ov

gemma-4-E4B-it-int8-ov

Superseded (2026-10-11). Intel re-exported OpenVINO/gemma-4-E4B-it-int8-ov on 2026-09-17 with fused attention, so prefix caching works there too. On an Arc Pro B60 with OpenVINO 2026.4.1 I get identical answers and the same turn times from both. Intel's loads faster - 17 s against 23-26 s. Use theirs. It needs OpenVINO 2026.4 or newer. This repo stays up so existing setups keep working, but no further updates. The rest is the original note.

OpenVINO INT8 export of google/gemma-4-E4B-it, re-exported so that prefix (KV) caching works.

Why this exists

Intel's published export, OpenVINO/gemma-4-E4B-it-int8-ov, cannot use openvino_genai's continuous-batching backend, so it gets no prefix caching at all:

No ScaledDotProductAttention operation observed in the graph,
cannot perform the SDPAToPagedAttention transformation.

That backend is built by rewriting fused ScaledDotProductAttention nodes into PagedAttention. Intel's language model was traced with decomposed attention, so there is nothing to rewrite. The tell is static — no GPU, no model load:

grep -c 'ScaledDotProductAttention' openvino_language_model.xml
#   OpenVINO/gemma-4-E4B-it-int8-ov  -> 0   (42 SoftMax instead)
#   this repo                        -> 42  (one per layer)

Its sibling exports (gemma-4-E2B-it-int4-ov, gemma-4-26b-a4b-it-int4-ov) are fine, so this is a per-build defect rather than anything about Gemma 4. optimum-intel pins the attention implementation only for models listed in FORCE_ATTN_MODEL_CLASSES (phi3_v, gemma2, llama4); gemma4 is not among them, so the export environment decides.

What changes, and what doesn't

Quality is unchanged. Against Intel's export on an identical probe set (OCR of small text incl. an arbitrary serial, a 3x3 letter-grid read, object counting, digits, colours/shapes, a text control), this export answered 7/7 byte-identically — including both of Intel's wrong answers. It adds a capability; it changes nothing else. Same 7.8 GB, same 42 layers, same 84 KB/token KV, same INT8 asymmetric channel-wise weights.

What you gain, measured on an Intel Arc Pro B60 (24 GB, OpenVINO 2026.3 release) with a ~7.9k-token repeated prefix — the shape of an agent's fixed system prompt:

Note the honest shape: Intel's export is faster cold, because the continuous-batching path prefills more slowly. This one pays a ~3.3 s premium once and is then ~2.6x faster per turn, breaking even by turn 3. If your client prewarms, the cold turn is paid before the first request and every user-visible turn is the fast one. For single-shot vision, Intel's export is the better choice; for agent loops, this one.

How it was made

pip install "optimum-intel[openvino] @ git+https://github.com/huggingface/optimum-intel.git"
pip install "transformers==5.5.4"        # last, so it overrides optimum's pin

optimum-cli export openvino \
  --model google/gemma-4-E4B-it \
  --task image-text-to-text \
  --weight-format int8 \
  gemma-4-E4B-it-int8-ov

Recorded in the IR: OpenVINO 2026.3.0-22451, optimum-intel 2.2.0.dev0+e1de5be, transformers 5.5.4. Export took ~3 minutes once the weights were local.

One gotcha if you reproduce this. The chat template is baked into openvino_tokenizer.xml at export time, so editing chat_template.jinja afterwards does nothing. Google's template uses Python-style implicit string concatenation inside raise_exception(...) — valid in Jinja2, rejected by openvino_genai's C++ Jinja parser with "Expected closing parenthesis in call args" at warmup. Replace the template in the source directory before exporting. The template here is the runtime-parseable one from Intel's export, with thanks.

Usage

import openvino_genai as ovg
sc = ovg.SchedulerConfig(); sc.enable_prefix_caching = True; sc.cache_size = 4
pipe = ovg.VLMPipeline("gemma-4-E4B-it-int8-ov", "GPU", scheduler_config=sc)
print(pipe.generate("What is in this image?", images=[img], max_new_tokens=100))

Served here through NoLlama, where prefix caching is on by default for GPU/CPU VLM slots.

Licence

Gemma. Use of this model is governed by the Gemma Terms of Use. This repository redistributes a quantized conversion of google/gemma-4-E4B-it; all rights and restrictions of the original apply unchanged.