gemma-4-12B-it-qat-lean-4bit-mlx

Gemma-4-12B-it (LEAN QAT, uniform 4-bit)

What this is

A lean MLX build of Google's Gemma4 quantization-aware-trained (QAT) checkpoint, converted directly from the QAT bf16 source weights with mlx-lm's own model-default quantization predicate -- not a third-party mixed-precision recipe. The MLP projections that most published MLX Gemma4 QAT builds silently upgrade to 8-bit stay at 4-bit / group-size-64 here (see Quantization policy below for this model's exact predicate, including whether it has an MoE router exception). The result is QAT quality at post-training-quantization (PTQ) size: accuracy calibrated for 4-bit weights, at the size a plain 4-bit build should actually be.

This is a LEAN build: the point is that the weights are exactly as small as the declared quantization policy promises, verifiable from config.json alone (see Validation below).

Provenance

• Source checkpoint: google/gemma-4-12B-it-qat-q4_0-unquantized, revision a89c069a80c767b0d378c4806b2953ae9d2c711d
• Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, programmatic API)
• Quantization policy: uniform 4-bit group-64, zero per-tensor exceptions
• Conversion script: scripts/convert_qat_lean.py (infinite repo, Yooz Labs) -- QUANT_POLICY["12b"]

Size

All sizes are on-disk decimal GB (safetensors shard bytes on disk); bits/weight is the effective average over all quantized tensors (packed weight bytes plus scale/bias bytes, divided by total parameter count), computed by scripts/convert_qat_lean.py verify.

Runtime memory (measured, not estimated)

Community build: mlx-community/gemma-4-12B-it-qat-4bit. Measured by the benchmark harness during real retrieval runs (peak of the serving process across all tasks per context length) on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Weight quantization pays off at runtime, not just on disk: every tensor a mixed-precision build silently upgrades to 8-bit stays resident at 8-bit.

Validation

Long-context retrieval A/B against mlx-community/gemma-4-12B-it-qat-4bit on the same stack (Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16): needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio.

n=10 per cell; * = adjudicated at n=30. Quality parity with the community QAT build. The FWE cells were adjudicated at n=30 (0.81 vs 0.72 at 8K, 0.56 vs 0.60 at 32K); the remaining deltas are within the n=10 noise band established during validation.

scripts/convert_qat_lean.py verify --path <this build> --model 12b confirms the quantization block in config.json carries no per-tensor overrides beyond uniform 4-bit group-64, zero per-tensor exceptions.

Usage

Loading note (mlx-lm <= 0.31.3): this build keeps its true model_type: gemma4_unified, which stock mlx-lm does not register until ml-explore/mlx-lm#1386 merges. Until then, alias it before loading:

import mlx_lm.utils as u
u.MODEL_REMAPPING["gemma4_unified"] = "gemma4"

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/gemma-4-12B-it-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where weight footprint is the budget that decides how much context fits on a real machine -- which is why the lean conversion exists.

Questions, issues, or corrections: [email protected]

License and attribution

Apache 2.0, inherited from the source checkpoint google/gemma-4-12B-it-qat-q4_0-unquantized (Google's Gemma4 QAT release; license verified apache-2.0 on the google/ repos). This is an independent, unofficial conversion; it is not distributed, endorsed, or certified by Google.