gemma-4-E4B-it-qat-lean-4bit-mlx

Gemma-4-E4B-it (LEAN QAT, uniform 4-bit)

What this is

A lean MLX build of Google's Gemma4 quantization-aware-trained (QAT) checkpoint, converted directly from the QAT bf16 source weights with mlx-lm's own model-default quantization predicate -- not a third-party mixed-precision recipe. The MLP projections that most published MLX Gemma4 QAT builds silently upgrade to 8-bit stay at 4-bit / group-size-64 here (see Quantization policy below for this model's exact predicate, including whether it has an MoE router exception). The result is QAT quality at post-training-quantization (PTQ) size: accuracy calibrated for 4-bit weights, at the size a plain 4-bit build should actually be.

This is a LEAN build: the point is that the weights are exactly as small as the declared quantization policy promises, verifiable from config.json alone (see Validation below).

Provenance

• Source checkpoint: google/gemma-4-E4B-it-qat-q4_0-unquantized, revision 223168fd2a569bf89aba76a0840b7970fbc9b621
• Conversion tool: mlx-lm 0.31.3 (mlx_lm.convert, programmatic API)
• Quantization policy: uniform 4-bit group-64, zero per-tensor exceptions
• Conversion script: scripts/convert_qat_lean.py (infinite repo, Yooz Labs) -- QUANT_POLICY["e4b"]

Size

All sizes are on-disk decimal GB (safetensors shard bytes on disk); bits/weight is the effective average over all quantized tensors (packed weight bytes plus scale/bias bytes, divided by total parameter count), computed by scripts/convert_qat_lean.py verify.

Runtime memory (measured, not estimated)

Community build: mlx-community/gemma-4-e4b-it-qat-OptiQ-4bit. Measured by the benchmark harness during real retrieval runs (peak of the serving process across all tasks per context length) on Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16. Weight quantization pays off at runtime, not just on disk: every tensor a mixed-precision build silently upgrades to 8-bit stays resident at 8-bit.

Validation

Long-context retrieval A/B against mlx-community/gemma-4-e4b-it-qat-OptiQ-4bit on the same stack (Apple M4 Pro (64 GB unified memory), macOS 26.5, mlx-lm 0.31.3, KV cache f16): needle-in-a-haystack (NIAH) and RULER-style tasks scored per-word, MRCR scored by sequence ratio.

n=10 per cell. Parity or better on every task at both context lengths: the lean build gives up nothing measurable against the mixed-precision community build.

scripts/convert_qat_lean.py verify --path <this build> --model e4b confirms the quantization block in config.json carries no per-tensor overrides beyond uniform 4-bit group-64, zero per-tensor exceptions.

Usage

from mlx_lm import load, generate

model, tokenizer = load("YoozLabs/gemma-4-E4B-it-qat-lean-4bit-mlx")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "List the first five prime numbers."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=64, verbose=True))

About Yooz Labs

Maintained by Yooz Labs -- sovereign, on-device AI for macOS. Everything we ship runs locally; nothing goes to a cloud. This build comes out of our long-context research line, where weight footprint is the budget that decides how much context fits on a real machine -- which is why the lean conversion exists.

Questions, issues, or corrections: [email protected]

License and attribution

Apache 2.0, inherited from the source checkpoint google/gemma-4-E4B-it-qat-q4_0-unquantized (Google's Gemma4 QAT release; license verified apache-2.0 on the google/ repos). This is an independent, unofficial conversion; it is not distributed, endorsed, or certified by Google.