Qwen3.5-9B-oQ4e-fp16-text-only

Qwen3.5-9B-oQ4e-fp16-text-only

oQ4e quantization of Qwen/Qwen3.5-9B with a float16 base dtype, built with craquehouse/model-lab.

Why fp16

oQ defaults to a bfloat16 base. M1 and M2 have no native bf16 path in the GPU, so this build uses fp16 instead — worth roughly 20% of prefill on those parts by oMLX's own reckoning. Both are 16-bit, so it is not a size trade. On M3 and later, hardware bf16 closes the gap and this build has no advantage over a normal one.

fp16's exponent range is narrower than bf16's (max ~65504 against ~3.4e38). If you see inf or degenerate output on hardware where the bf16 build is fine, that is the thing to suspect.

The vision and audio towers are dropped entirely, which is what makes this build genuinely fp16: oQ holds towers at fp32 whenever the target dtype is float16, so a multimodal build of the same model is not the dtype its name claims.

Tensors written: F16 x653, F32 x24, U32 x250. The 24 fp32 tensors are all A_log, the SSM / linear-attention state parameters, which oQ never quantizes or narrows.

Reasoning is on by default

This is a thinking model and reasoning is enabled unless you turn it off — the reply arrives in reasoning_content, and content can come back empty if max_tokens runs out mid-thought.

That default comes from this repo's own chat template, which gates on the variable being explicitly false:

{%- if enable_thinking is defined and enable_thinking is false %}
    {{- '<think>\n\n</think>\n\n' }}
{%- else %}
    {{- '<think>\n' }}

An unset enable_thinking takes the else branch and opens a thinking block, so silence means reasoning. Turn it off per request:

{"chat_template_kwargs": {"enable_thinking": false}}

A top-level "enable_thinking": false does not work — oMLX's request model does not declare that field, so it is silently dropped. reasoning_effort is accepted and then discarded, as this template has no such variable.

The template here is byte-identical to Qwen3.5-4B's (sha256 a4aee8af…, 7756 bytes), so this behaviour is the family's rather than this size's. It was confirmed on this build rather than assumed: a 128-token generation from the unconfigured model spent its whole budget inside <think> and returned empty content.

Turning it off for clients that cannot send that

Home Assistant, and most OpenAI-compatible integrations, give you no way to set chat_template_kwargs. Set the default on the server instead, per model, in ~/.omlx/model_settings.json:

{
  "version": 1,
  "models": {
    "Qwen3.5-9B-oQ4e-fp16-text-only": { "enable_thinking": false }
  }
}

The key is the API-visible model id, which is the model directory's basename — not an org-qualified path, even when the model is stored under one. So the key must match whatever you named the directory you serve from: the assist-mini numbers below were taken with it served as qwen35-9b-oq4e, and that is the key that was set. Get this wrong and nothing errors; you simply measure a thinking model. oMLX reads this file at startup and does not rewrite it.

This sets a default, not a lock: a client that does send chat_template_kwargs can still turn reasoning back on. oMLX's ModelSettings also carries a forced_ct_kwargs field, described as keys that cannot be overridden per request; that path is untested here.

Recipe

python recipes/oq_quantize.py \
    --model Qwen/Qwen3.5-9B \
    --level 4 --enhanced --text-only \
    --dtype float16 \
    --out Qwen3.5-9B-oQ4e-fp16-text-only

Built with oMLX 0.6.3rc2. The version is part of the recipe, not trivia: 0.5.0-rc1 shipped an oQ4/oQ4e regression worth ~37 points of GSM8K, and the checkpoints loaded and generated fluently throughout (#2172).

Allocation

Base 4-bit affine at group 64, with 80 tensors held above it by oQ's sensitivity budget: 74 at 5-bit, 6 at 6-bit. The boosts are the point of oQ and they are invisible in the file size — build.json in this repo records them, and oq_imatrix_report.json records what the 128-sample calibration pass actually covered.

Measured

Throughput and peak memory

Measured by oMLX 0.6.3rc2's own benchmark (/admin/api/bench/start, corpus code_python, TG=128) with reasoning disabled, driven by harness/omlx_bench.py in the model-lab repo so the numbers are reproducible rather than screenshotted.

Single request

Prefill holds 470–480 t/s to 8k and falls to 422 t/s at 16k. Decode goes 45.3 to 39.5 t/s across a 16x context increase — -13%, with TPOT rising 22.3 to 25.5 ms. GPU utilisation is 98.7% at PP1024 and ≥99.7% everywhere above it, so none of these rows is limited by anything but the GPU.

Concurrent (PP1024)

3.8x the aggregate throughput at 4x concurrency, with per-request latency degrading roughly in proportion. Peak memory is flat at 6.09 GiB across all three: KV for four 1024-token streams is negligible against the weights.

oMLX pins concurrent tests to PP1024 — it generates batch prompts at that length regardless of what else you ask for — so this says nothing about concurrency at long context.

Peak is the number that decides whether this fits

Weights are 5.31 GB. Peak is 6.09 GiB at 1k context and 7.87 GiB at 16k — so budget roughly 1.5–2.5 GiB above the file size, growing with context, and size alone will mislead you by that much.

On a 16 GB machine that is about half of RAM at long context, which is usable but leaves less room for a co-resident audio stack than the 4B build does, at 3.50 GiB / 5.36 GiB for the same two context lengths. If you are sharing memory with a speech pipeline, that difference is the thing to weigh — not the 2.8 GB difference in file size.

Those 4B figures come from a separate run with different neighbours resident, so the gap is indicative rather than a paired measurement. Both builds would have to be benched back to back on one machine to make it a number worth quoting to two decimal places.

What these numbers are not

• Measured on a 48 GiB machine, not the 16 GB one the sizing advice above is aimed at. oMLX's memory guard and cache sizing behave differently under real pressure, so treat the peaks as a floor for a 16 GB host rather than a prediction.
• The server held four models in its catalog. Peak is process-wide, so these figures include anything else oMLX had resident. They are not best-case.
• One run per cell. No repeats, no confidence intervals. Directional.

Home Assistant capability

90.3% on assist-mini (177/196), verified by asserting entity-state changes inside Home Assistant 2026.7.4 — not by matching strings against expected output.

Served by oMLX 0.6.3rc2, reasoning disabled. 49 unique tasks repeated 4x = 196 records. Repeats of one task are correlated, so treat 196 as a record count and not as an independent n -- a confidence interval computed as though it were is too narrow. Treat a single category as directional and the overall figure as the comparable one.

Divergence from the source

Teacher-forced against Qwen/Qwen3.5-9B, 64 x 1024-token chunks per corpus. Lower is closer to the source; 0 would be identical. Measured on one corpus only, so this is not a bound on general prose, where a quantization has more room to diverge.

What is not measured

Divergence says how far this build's output distribution moved from the checkpoint it was quantized from. It does not say how the model scores on any task — there is no MMLU, no GSM8K, no HumanEval here, and none is implied. Those measure how good the model is; a quantization's claim is that it is the same model, which is what the number above tests.

Licence

Inherits the licence of the base model, Qwen/Qwen3.5-9B. Quantization does not create a new work; check the base model's card before redistributing.