gemma-4-E2B-it-qat-oQ4e-fp16-text-only

gemma-4-E2B-it-qat-oQ4e-fp16-text-only

oQ4e quantization of google/gemma-4-E2B-it-qat-q4_0-unquantized with a float16 base dtype, built with craquehouse/model-lab.

Why fp16

oQ defaults to a bfloat16 base. M1 and M2 have no native bf16 path in the GPU, so this build uses fp16 instead — worth roughly 20% of prefill on those parts by oMLX's own reckoning. Both are 16-bit, so it is not a size trade. On M3 and later, hardware bf16 closes the gap and this build has no advantage over a normal one.

fp16's exponent range is narrower than bf16's (max ~65504 against ~3.4e38). If you see inf or degenerate output on hardware where the bf16 build is fine, that is the thing to suspect.

This build is fp16 throughout — no bf16 and no fp32 tensors, only fp16 and packed 4-bit weights. The vision and audio towers are dropped entirely, which is both what makes that true (oQ holds towers at fp32 for fp16 targets) and why it is 2 GB smaller than the multimodal build.

Note that dropping the towers also shifted the language model's own bit allocation: they were counted into oQ's bits-per-weight budget while never being quantized, so removing them cost five tensors a bit level. See the write-up.

Reasoning is on by default

This is a thinking model, and on the mlx-lm path reasoning is enabled unless you turn it off — the reply arrives in reasoning_content, and content can come back empty if max_tokens runs out mid-thought.

That is not the chat template's default, which is false. It comes from mlx-lm's TokenizerWrapper:

if "enable_thinking" not in kwargs:
    kwargs["enable_thinking"] = self.has_thinking   # True: think_token is set

Gemma 4 declares think_token in tokenizer_config.json (upstream Google's, not this build's doing), so the wrapper fills in True whenever the caller is silent. Turn it off per request:

{"chat_template_kwargs": {"enable_thinking": false}}

That drops the prompt by 7 tokens and returns the answer in content. A top-level "enable_thinking": false does not work — oMLX's request model does not declare that field, so it is silently dropped. reasoning_effort is accepted but the Gemma 4 template has no such variable, so it is discarded too.

Turning it off for clients that cannot send that

Home Assistant, and most OpenAI-compatible integrations, give you no way to set chat_template_kwargs. Set the default on the server instead, per model, in ~/.omlx/model_settings.json:

{
  "version": 1,
  "models": {
    "gemma-4-E2B-it-qat-oQ4e-fp16-text-only": { "enable_thinking": false }
  }
}

or from oMLX's admin panel under per-model settings, which applies without a restart. A bare request then renders 19 prompt tokens and answers in content, and a client that does send chat_template_kwargs can still turn reasoning back on. Add "forced_ct_kwargs": ["enable_thinking"] if you want the server's choice to win regardless.

There is no global switch — SchedulerConfig.enable_thinking exists in oMLX's settings schema but is never read. It has to be set per model.

The multimodal sibling loads through mlx-vlm, which does not apply that wrapper, and so defaults the other way. Same model, opposite default.

Recipe

python recipes/oq_quantize.py \
    --model google/gemma-4-E2B-it-qat-q4_0-unquantized \
    --level 4 --enhanced --text-only \
    --dtype float16 \
    --out gemma-4-E2B-it-qat-oQ4e-fp16-text-only

Built with oMLX 0.6.3rc2. The version is part of the recipe, not trivia: 0.5.0-rc1 shipped an oQ4/oQ4e regression worth ~37 points of GSM8K, and the checkpoints loaded and generated fluently throughout (#2172).

Allocation

Base 4-bit affine at group 64, with 98 tensors held above it by oQ's sensitivity budget: 73 at 5-bit, 13 at 6-bit, 12 at 8-bit. The boosts are the point of oQ and they are invisible in the file size — build.json in this repo records them, and oq_imatrix_report.json records what the 128-sample calibration pass actually covered.

Measured

Divergence from the source

Teacher-forced against google/gemma-4-E2B-it-qat-q4_0-unquantized, 64 x 1024-token chunks per corpus. Lower is closer to the source; 0 would be identical. Home Assistant traffic is structured and low-entropy, so the quantization has less room to disagree there — the general-prose row is the pessimistic bound.

Behaviour

Greedy-decoded over 200 Home Assistant prompts and compared against the source's own completions: 71.5% byte-identical, 8.8 tokens agreed before the first divergence, and 0 degenerate completions (loops or empty output).

That last number is the one that matters — it is the failure an fp16 exponent overflow would produce, and it is zero here. The agreement rate is a gate rather than a score: at n=200 it cannot separate builds, and most of the disagreement is prompts that every quantization finds hard rather than damage specific to this one.

What is not measured

Divergence says how far this build's output distribution moved from the checkpoint it was quantized from. It does not say how the model scores on any task — there is no MMLU, no GSM8K, no HumanEval here, and none is implied. Those measure how good the model is; a quantization's claim is that it is the same model, which is what the number above tests.

Licence

Inherits the licence of the base model, google/gemma-4-E2B-it-qat-q4_0-unquantized. Quantization does not create a new work; check the base model's card before redistributing.