Qwen3.5-4B-oQ4e-fp16-text-only

Qwen3.5-4B-oQ4e-fp16-text-only

oQ4e quantization of Qwen/Qwen3.5-4B with a float16 base dtype, built with craquehouse/model-lab.

Why fp16

oQ defaults to a bfloat16 base. M1 and M2 have no native bf16 path in the GPU, so this build uses fp16 instead — worth roughly 20% of prefill on those parts by oMLX's own reckoning. Both are 16-bit, so it is not a size trade. On M3 and later, hardware bf16 closes the gap and this build has no advantage over a normal one.

fp16's exponent range is narrower than bf16's (max ~65504 against ~3.4e38). If you see inf or degenerate output on hardware where the bf16 build is fine, that is the thing to suspect.

The vision and audio towers are dropped entirely, which is what makes this build genuinely fp16: oQ holds towers at fp32 whenever the target dtype is float16, so a multimodal build of the same model is not the dtype its name claims.

Tensors written: F16 x651, F32 x24, U32 x249. The 24 fp32 tensors are all A_log, the SSM / linear-attention state parameters, which oQ never quantizes or narrows.

Reasoning is on by default

This is a thinking model and reasoning is enabled unless you turn it off — the reply arrives in reasoning_content, and content can come back empty if max_tokens runs out mid-thought.

That default comes from this repo's own chat template, which gates on the variable being explicitly false:

{%- if enable_thinking is defined and enable_thinking is false %}
    {{- '<think>\n\n</think>\n\n' }}
{%- else %}
    {{- '<think>\n' }}

An unset enable_thinking takes the else branch and opens a thinking block, so silence means reasoning. Turn it off per request:

{"chat_template_kwargs": {"enable_thinking": false}}

Note this costs 2 prompt tokens rather than saving any — the disabled form emits a pre-closed <think>\n\n</think>\n\n pair, which is longer than the bare opener. Measured on the same prompt: 16 prompt tokens and 27 output in 0.71s with reasoning off, against 14 prompt tokens and 224 output in 3.82s with it on.

A top-level "enable_thinking": false does not work — oMLX's request model does not declare that field, so it is silently dropped. reasoning_effort is accepted and then discarded, as this template has no such variable. Both were tested: each returned a full reasoning block.

Turning it off for clients that cannot send that

Home Assistant, and most OpenAI-compatible integrations, give you no way to set chat_template_kwargs. Set the default on the server instead, per model, in ~/.omlx/model_settings.json:

{
  "version": 1,
  "models": {
    "Qwen3.5-4B-oQ4e-fp16-text-only": { "enable_thinking": false }
  }
}

The key is the API-visible model id, which is the model directory's basename — not an org-qualified path, even when the model is stored under one. oMLX reads this file at startup and does not rewrite it.

This sets a default, not a lock: a client that does send chat_template_kwargs can still turn reasoning back on, which the measurements above demonstrate — they were taken with this file in place. oMLX's ModelSettings also carries a forced_ct_kwargs field, described as keys that cannot be overridden per request; that path is untested here.

Recipe

python recipes/oq_quantize.py \
    --model Qwen/Qwen3.5-4B \
    --level 4 --enhanced --text-only \
    --dtype float16 \
    --out Qwen3.5-4B-oQ4e-fp16-text-only

Built with oMLX 0.6.3rc2. The version is part of the recipe, not trivia: 0.5.0-rc1 shipped an oQ4/oQ4e regression worth ~37 points of GSM8K, and the checkpoints loaded and generated fluently throughout (#2172).

Allocation

Base 4-bit affine at group 64, with 98 tensors held above it by oQ's sensitivity budget: 91 at 5-bit, 7 at 6-bit. The boosts are the point of oQ and they are invisible in the file size — build.json in this repo records them, and oq_imatrix_report.json records what the 128-sample calibration pass actually covered.

Measured

Throughput

Measured by oMLX 0.6.3rc2's own benchmark (/admin/api/bench/start, corpus code_python, TG=128) on an Apple M2 Pro with 16 GB, reasoning disabled.

Single request

Prefill settles around 420-440 t/s once the GPU saturates; the PP1024 row is flattered by running at 92% utilisation against ~99.7% everywhere else. Decode falls 59.6 to 48.4 t/s across a 16x context increase — -19%, with TPOT rising 16.9 to 20.8 ms.

Concurrent (PP1024)

2.5x the aggregate throughput at 4x concurrency, with per-request latency degrading roughly in proportion. Peak memory is flat at 3.50 GiB across all three: KV for four 1024-token streams is negligible against the weights.

oMLX pins concurrent tests to PP1024 — it generates batch prompts at that length regardless of what else you ask for — so this says nothing about concurrency at long context.

One run per cell, on a machine also serving an embedding model and a separate audio stack, so the memory figures include a co-resident neighbour and none of these are best-case. Treat them as directional.

Home Assistant capability

86.7% on assist-mini (170/196), verified by asserting entity-state changes inside Home Assistant 2026.7.4 — not by matching strings against expected output.

Served by oMLX 0.6.3rc2, reasoning disabled. 49 unique tasks repeated 4x = 196 records. Repeats of one task are correlated, so treat 196 as a record count and not as an independent n -- a confidence interval computed as though it were is too narrow. Treat a single category as directional and the overall figure as the comparable one.

Divergence from the source

No divergence measurement was run for this build. See model-lab.

What is not measured

Divergence says how far this build's output distribution moved from the checkpoint it was quantized from. It does not say how the model scores on any task — there is no MMLU, no GSM8K, no HumanEval here, and none is implied. Those measure how good the model is; a quantization's claim is that it is the same model, which is what the number above tests.

Licence

Inherits the licence of the base model, Qwen/Qwen3.5-4B. Quantization does not create a new work; check the base model's card before redistributing.