gemma-4-12B-it-W4A16-FP8KV

gemma-4-12B-it-W4A16-FP8KV

Quantization of google/gemma-4-12B-it built for Ampere GPUs: INT4 group-128 weights, bfloat16 activations and a calibrated FP8 KV cache. Produced with llm-compressor 0.13.0 (with a custom compressed_tensors 0.18.0 patch), shipped in the compressed-tensors pack-quantized format. For model capabilities, benchmarks and usage documentation see the base model card.

[!NOTE] To run this model on Ampere, or any GPU earlier than SM100, you must serve it from a vLLM build carrying the serve fixes described below. The zankich/vllm fork integrates them on its default branch, and zankich/vllm-openai:latest on Docker Hub is the prebuilt serve image from that fork. On SM100+ stock vLLM serves this checkpoint as-is with no patches.

Running the model

Two serving modes on pre-SM100 GPUs, one image, chosen by flags.

Text-only, FlashInfer

docker run -d --gpus all -p 8000:8000 \
  -v /path/to/gemma-4-12B-it-W4A16-FP8KV:/models/gemma-4-12B-it-W4A16-FP8KV:ro \
  zankich/vllm-openai:latest \
  /models/gemma-4-12B-it-W4A16-FP8KV \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --attention-backend FLASHINFER \
  --attention-config '{"use_trtllm_attention": false}' \
  --language-model-only

--language-model-only is required on this path: FlashInfer does not support the multimodal attention of the underlying Gemma4Unified architecture, so the audio and vision paths cannot serve here and only waste memory. Text-only serving with the calibrated E4M3 scales is the configuration this checkpoint was built on.

Multimodal, Triton

docker run -d --gpus all -p 8000:8000 \
  -v /path/to/gemma-4-12B-it-W4A16-FP8KV:/models/gemma-4-12B-it-W4A16-FP8KV:ro \
  zankich/vllm-openai:latest \
  /models/gemma-4-12B-it-W4A16-FP8KV \
  --dtype bfloat16 \
  --kv-cache-dtype fp8_e5m2 \
  --attention-backend TRITON_ATTN \
  --max-num-batched-tokens 3072

Both vision and audio inputs serve on this path. --max-num-batched-tokens must exceed 2496: one image is 2496 encoder tokens and smaller budgets crash the multimodal budget check at startup. The image ships the vllm[audio] extras, so audio input needs no extra installation.

The FlashInfer mode serves the calibrated E4M3 scales as shipped with --language-model-only. The Triton mode serves vision and audio with the KV cache as E5M2 — the same scales but roughly twice the per-element quantization noise, because Triton cannot compile the E4M3 dtype below SM89. Verified on the Triton path: image inputs answered exactly (shape and color correct on a synthetic test image), and a 20-second real-speech clip transcribed with correct semantic content at roughly 80-85% word accuracy, the errors consistent with W4A16 weights plus the E5M2 KV path on accented archive audio. No logprob-level comparison against E4M3 has been run.

One interaction worth knowing when serving audio: verbatim transcription requests should disable thinking (chat_template_kwargs: {"enable_thinking": false} at request level). With thinking enabled, transcription requests tend to transcribe inside the reasoning channel and exhaust the token budget before producing content. Analysis-style audio requests behave normally with thinking on.

Summary

KV cache calibration

The KV cache scales are calibrated during the same forward passes as the GPTQ weights, via kv_cache_scheme on the modifier, so no separate calibration step runs. The unified 12B mixes two KV geometries: 40 sliding-window layers at 8 KV heads and head_dim 256, and 8 full-attention multi-query layers at 1 KV head and head_dim 512. The checkpoint carries 96 k_scale/v_scale tensors, one pair per layer, each sized for its own geometry. Serve with --kv-cache-dtype fp8 so vLLM loads them.

[!WARNING] Serve with the calibrated FP8 scales, not bfloat16 KV. Bfloat16 KV silently corrupts generation while the server reports healthy: greedy output degrades to garbage and every structured probe fails, with nothing in the logs. The calibrated scales are part of the checkpoint's contract, not a tuning option.

What is quantized

Every Linear except the ignore list below: all attention and FFN projections across both attention geometries.

What is kept at bf16

• lm_head
• All text embeddings, and the audio and vision embedding projections, matching llm-compressor's official gemma-4 recipe. The unified 12B checkpoint carries no vision or audio tower weights, only the embedding projections, so there is no tower to preserve.

The ignore-list shape as executed:

KV_CACHE_SCHEME = {
    "num_bits": 8,
    "type": "float",
    "strategy": "tensor",
    "dynamic": False,
    "symmetric": True,
}

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "lm_head",
        "re:.*vision.*",
        "re:.*audio.*",
        "re:.*embed.*",
    ],
    kv_cache_scheme=KV_CACHE_SCHEME,
)

As executed the run also used dampening_frac: 0.2. The full recipe ships as recipe.yaml.

Calibration data

512 samples at max sequence length 2048, a chat-agent mix of interactive coding turns and function-level public code in Python, TypeScript, Go and shell. The calibration set is built from private session data and is not published. No accuracy benchmark delta against the bf16 base is published for this checkpoint.

Serving below SM100

Both fixes are on the in the zankich/vllm fork, whose README documents them. zankich/vllm-openai:latest on Docker Hub is the prebuilt serve image.

What the fixes cover for this model: the unified 12B's full-attention layers are large-head (head_dim 512), and FlashInfer gates one-byte-KV large-head modules to SM100+, so FP8 KV needs the widened SM8 opt-in; the Triton fix serves the multimodal path below SM100 with the KV cache as e5m2, calibrated scales loading unchanged. The FlashInfer lines are required together on anything earlier than SM100: the default attention path selects a trtllm-gen variant that is SM100+ only.

Producing the quant (compressed-tensors)

The compressed-tensors-heterogeneous-kv.patch in this repo is needed only to produce the quant — it patches llm-compressor's compressed-tensors dependency, not vLLM. The quant was built with llm-compressor 0.13.0, which installs compressed-tensors 0.18.0, and the patch is written against exactly that version. The unified 12B mixes two attention geometries in one model, Transformers 5 exposes that as a per_layer_config list, and stock compressed-tensors 0.18.0 hands every attention module the global text config, so get_num_kv_heads raises AmbiguousGlobalPerLayerAttributeError before any quantization runs. The patch resolves each module's own per_layer_config slice by layer index. Homogeneous models take the unchanged global path, so it is inert for everything except heterogeneous models. Check upstream first: a release carrying per-layer config resolution in initialize_hooked_kv_cache makes it obsolete.

Chat template

The repo ships the official chat_template.jinja, verbatim from the base model. vLLM and transformers auto-detect a chat_template.jinja file next to the weights, so this is what you get by default.