gemma-4-31b-it-qat-q4-apml

gemma-4-31b-it-qat-q4.apml — Apertura model bundle (lattice-exact QAT)

Replaced 2026-10-09. Revisions before this date held an MLX-affine group-64 quantization of the QAT weights (28% bit-exact, 95% top-1 vs bf16). This revision is the lattice-exact recipe below; the bundle layout and name are unchanged.

Gemma 4 31B instruction-tuned, Google's QAT-Q4 weights converted to the Apertura .apml bundle format with the lattice-exact recipe: MLX-affine 4-bit, group size 32, where each block's scale is the QAT-learned step itself (scale = d, bias = -8·d), so the 4-bit weights the kernel sees are the trained lattice points — not a second, misaligned quantization of them. The tied embedding / LM head sits on the same lattice and is stored exact at 4 bits. tokenizer.json + chat_template.jinja included, runtime mlx, architecture gemma4.

Consumed by Apertura — an Objective-C++ transformer engine + macOS chat app on MLX.

Why this recipe

google/gemma-4-31B-it-qat-q4_0-unquantized is bf16 weights that already sit on an int4 lattice: per 32-element block, w = bf16(k·d) with codes k ∈ [-8, 7] and a learned per-block step d. Running a min/max affine quantizer over that (as the earlier revisions of this repo, group 64, did) picks a different scale and zero point and lands most weights off their trained values — the same failure Unsloth documented for naive Q4_0 GGUFs of these checkpoints. Here d is recovered from the lattice structure per block and written as the affine scale.

Measured against the bf16 QAT checkpoint, over all 30.7 B quantized weights:

The 9.3% that are not bit-identical differ by one bf16 rounding step: the checkpoint stores bf16 roundings of k·d, so no single bf16 step reproduces every element. The bundle and the checkpoint are two equally faithful bf16 renderings of the trained lattice. Group 32 costs ~23% decode throughput vs group 64 on this model (Apertura roadmap §3); fidelity was the aim.

weights/mlx-q4/quantization.json records lattice: qat-int4-g32 and the fit statistics. Do not pass --quant-embed when loading: it would re-quantize the exact 4-bit head.

Download

hf download apocryphx/gemma-4-31b-it-qat-q4-apml --local-dir gemma-4-31b-it-qat-q4.apml

License

Gemma is provided under and subject to the Gemma Terms of Use. By downloading this model you agree to those terms, including the Gemma Prohibited Use Policy. This repository redistributes a quantized Model Derivative of google/gemma-4-31B-it-qat-q4_0-unquantized; the same terms and use restrictions apply to it and to any further derivatives.