gemma-4-E2B-it-qat-oQ4e-fp16-mtp

gemma-4-E2B-it-qat-oQ4e-fp16-mtp

oQ4e quantization of google/gemma-4-E2B-it-qat-q4_0-unquantized with a float16 base dtype, built with craquehouse/model-lab.

[!IMPORTANT] This checkpoint does not run on stock mlx-lm. It carries a grafted multi-token-prediction head under language_model.mtp.*, which mlx-lm has no binding site for. Loading fails outright:

ValueError: Received parameters not in model: language_model.mtp....

Serve it with oMLX, whose Gemma 4 Lightning MTP runtime patch attaches the head at load:

brew install jundot/omlx/omlx
omlx serve --model-dir /path/containing/this/model

If you want a build that loads anywhere, use the text-only variant craquehouse/gemma-4-E2B-it-qat-oQ4e-fp16-text-only instead. It has no MTP head, no vision or audio towers, and loads under stock mlx-lm.

Why fp16

oQ defaults to a bfloat16 base. M1 and M2 have no native bf16 path in the GPU, so this build uses fp16 instead — worth roughly 20% of prefill on those parts by oMLX's own reckoning. Both are 16-bit, so it is not a size trade. On M3 and later, hardware bf16 closes the gap and this build has no advantage over a normal one.

fp16's exponent range is narrower than bf16's (max ~65504 against ~3.4e38). If you see inf or degenerate output on hardware where the bf16 build is fine, that is the thing to suspect.

[!NOTE] The towers are fp32, not fp16. oQ deliberately holds vision and audio encoders at full precision whenever the target dtype is float16 — encoder stacks are where fp16's exponent range runs out. That is worth 0.94 GB here against the equivalent bf16 build. The language model is fp16 as intended; the grafted MTP head stays bf16, at its donor's dtype.

If you only need text, the text-only variant avoids all of this and is 2 GB smaller.

Reasoning is off by default

This is a thinking model, but this build loads through mlx-vlm, which does not apply mlx-lm's TokenizerWrapper — so reasoning is off unless asked for:

{"chat_template_kwargs": {"enable_thinking": true}}

Worth knowing because the text-only sibling defaults the other way: it takes the mlx-lm path, where TokenizerWrapper sets enable_thinking from the presence of think_token and overrides the template's own default(false). Two builds of one model, opposite defaults, same prompt.

A top-level "enable_thinking" is silently ignored either way — oMLX's request model does not declare the field. Use chat_template_kwargs.

Recipe

python recipes/oq_quantize.py \
    --model google/gemma-4-E2B-it-qat-q4_0-unquantized \
    --level 4 --enhanced --mtp-donor google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant \
    --dtype float16 \
    --out gemma-4-E2B-it-qat-oQ4e-fp16-mtp

Built with oMLX 0.6.3rc2. The version is part of the recipe, not trivia: 0.5.0-rc1 shipped an oQ4/oQ4e regression worth ~37 points of GSM8K, and the checkpoints loaded and generated fluently throughout (#2172).

Allocation

Base 4-bit affine at group 64, with 100 tensors held above it by oQ's sensitivity budget: 72 at 5-bit, 13 at 6-bit, 15 at 8-bit. The boosts are the point of oQ and they are invisible in the file size — build.json in this repo records them, and oq_imatrix_report.json records what the 128-sample calibration pass actually covered.

Measured

Divergence from the source

Teacher-forced against google/gemma-4-E2B-it-qat-q4_0-unquantized, 64 x 1024-token chunks per corpus. Lower is closer to the source; 0 would be identical. Home Assistant traffic is structured and low-entropy, so the quantization has less room to disagree there — the general-prose row is the pessimistic bound.

Behaviour

Greedy-decoded over 200 Home Assistant prompts and compared against the source's own completions: 75.0% byte-identical, 9.1 tokens agreed before the first divergence, and 0 degenerate completions (loops or empty output).

That last number is the one that matters — it is the failure an fp16 exponent overflow would produce, and it is zero here. The agreement rate is a gate rather than a score: at n=200 it cannot separate builds, and most of the disagreement is prompts that every quantization finds hard rather than damage specific to this one.

What is not measured

Divergence says how far this build's output distribution moved from the checkpoint it was quantized from. It does not say how the model scores on any task — there is no MMLU, no GSM8K, no HumanEval here, and none is implied. Those measure how good the model is; a quantization's claim is that it is the same model, which is what the number above tests.

Licence

Inherits the licence of the base model, google/gemma-4-E2B-it-qat-q4_0-unquantized. Quantization does not create a new work; check the base model's card before redistributing.