gemma-4-12B-it-W4A16

gemma-4-12B-it-W4A16

Int4 weight-only quantization of google/gemma-4-12B-it, in compressed-tensors format for vLLM. 7.74 GB, down from 23.92 GB.

Unofficial and unaffiliated with Google. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

If you want the best 4-bit Gemma 4 12B, use Google's own gemma-4-12B-it-qat-w4a16-ct instead. It is quantization-aware trained, this one is post-training round-to-nearest. This repository exists as a small, fast-loading test artifact and as a worked example of a fully reproducible data-free recipe.

What was changed

Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.

Left at bfloat16 — quantizing these produces a checkpoint vLLM cannot load, because it builds Gemma 4's non-text towers with quant_config=None:

• vision_tower, vision_embedder, embed_vision and the vision projections
• audio_tower, embed_audio and the audio projection
• embed_tokens (tied to the output head in this checkpoint) and lm_head

The 2.5 GB difference is almost entirely those two rows: Google's build materializes an untied lm_head (2.01 GB) and its group-32 scales cost ~0.5 GB more than group-128.

Reproducing this checkpoint

Built with llm-quantizer — one command:

./llmq.py run --profile gemma-4-12b-it

which is equivalent to:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="google/gemma-4-12B-it",   # re-sharded to 4 GB shards first
    save_directory="gemma-4-12B-it-W4A16",
    scheme="W4A16",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed_tokens.*"],
    max_workers=2,
    device="cuda:0",
)

The whole build ran on a 16 GB laptop GPU: model_free_ptq loads one shard at a time, so re-sharding the single 23.92 GB source file into 4 GB pieces puts the peak at ~5 GB.

Usage

vllm serve <this-repo> --max-model-len 65536

Do not pass --quantization; compressed-tensors is detected from config.json. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

Version note. Gemma 4 needs transformers >= 5.10, but vLLM ≤ 0.27.1 cannot serve Gemma 4 with transformers >= 5.15: head_dim became a per-layer attribute and older vLLM reads it globally, raising AmbiguousGlobalPerLayerAttributeError at engine-config time. Use either transformers < 5.15 with vLLM 0.25–0.27, or vLLM >= 0.28, which handles both layouts.

Evaluation

None. This checkpoint has not been benchmarked, and round-to-nearest quantization is expected to degrade quality more than the vendor's QAT build — how much, for your task, is unmeasured here. Treat published Gemma 4 benchmark numbers as describing the bf16 model, not this one.

License

Apache 2.0, inherited from the base model — see LICENSE and Google's Gemma 4 license page. The base repository ships no LICENSE file, so the Apache-2.0 text is included here for redistribution. "Gemma" is Google's mark; this repository is not endorsed by or affiliated with Google.