gemma-4-12b-it-AWQ-rtn

gemma-4-12b-it AWQ-RTN (int4, gs=32)

W4A16 round-to-nearest (RTN, mse, group_size=32, asymmetric) quantization of google/gemma-4-12b-it, built for vLLM on NVIDIA V100 (SM70) via the 1Cat-vLLM fork.

GSM8K 50q 5-shot: 96% (bf16 reference: 94%).

Why this checkpoint exists

Standard AWQ (with duo_scaling / activation smoothing) poisons Gemma-4: the smoothing scales (~×43) applied to Gemma's huge layernorm weights destroy fp16 headroom and the model emits all-pad output on content-rich prompts. This affects several community Gemma-4 AWQ uploads. RTN-with-mse-selection avoids the smoothing entirely and preserves quality.

Layernorm weights are bit-identical to the bf16 original (verified).

Update

Google has since released official QAT W4A16 compressed-tensors checkpoints (google/gemma-4-12B-it-qat-w4a16-ct), which are QAT-trained and likely higher quality. This RTN checkpoint remains useful as a smoothing-free baseline.

Usage

from vllm import LLM
llm = LLM(model="JelleFoks/gemma-4-12b-it-AWQ-rtn", dtype="float16",
          trust_remote_code=True)

Validated on 1Cat-vLLM sm70-v100-rebase-new: TG B=1 37.8 / B=8 186.7 tok/s, PP ctx=512 1698 tok/s (V100-32GB, 1380 MHz).