gemma-4-12b-it-qat-4bit-mlx

gemma-4-12b-it-qat-4bit-mlx

Uniform 4-bit MLX conversion of Google's QAT (quantization-aware trained) Gemma 4 12B instruction-tuned checkpoint. Built and used by ModelCaddy as the on-device generation model for its Pro tier (local conversation extraction and memory synthesis on Apple Silicon).

Provenance

• Source: google/gemma-4-12B-it-qat-q4_0-unquantized — the bf16 QAT weights Google publishes for downstream quantization.
• Conversion: mlx_vlm.convert -q --q-bits 4 --q-group-size 64 (mlx-vlm 0.6.13, mlx 0.32.0).
• Result: ~6.3 GB on disk, 4.51 effective bits per weight, uniform 4-bit affine quantization (group size 64). No layers held back at higher precision — the QAT training is what preserves quality at uniform 4-bit.

Why the Hub shows "~2B params" for a 12B model

The Hub counts stored safetensors elements. MLX packs eight 4-bit weights into each stored U32, so the counter sees 1.49B U32 containers + 0.38B BF16 quantization scales ≈ 1.9B stored elements and rounds to "2B". Unpacked, the model is 12.33B parameters (1.49B × 8 + 0.38B) — the full Gemma 4 12B. Cross-check: 12.33B × 4.51 bits ÷ 8 ≈ 6.9 GB, matching the shard sizes. Every 4-bit MLX repo on the Hub under-displays the same way.

Requirements

Apple Silicon and mlx-vlm >= 0.6.x (the gemma4_unified architecture is not supported by mlx-vlm 0.5.x).

pip install -U mlx-vlm
python -m mlx_vlm generate --model modelcaddy/gemma-4-12b-it-qat-4bit-mlx \
  --prompt "Summarize: ..." --max-tokens 256

License

Gemma is provided under and subject to the Gemma Terms of Use. This repository redistributes a quantized derivative of Google's Gemma 4 weights; use is governed by those terms and the Gemma Prohibited Use Policy.