Qwen3.8-27B-AWQ-5.0bpw-Sharp

Qwen3.8-27B — AWQ 5.0bpw (group size 64, multimodal calibration)

AWQ-calibrated quantization of Qwen/Qwen3.8-27B (27.8B dense, 64 layers, hybrid GatedDeltaNet + full attention, 256K context, MTP head, vision tower) for Apple Silicon via oMLX.

17.36 GB · BPW 5.00 · 3.2× smaller than bf16

A group-size-64 variant of Qwen3.8-27B-AWQ-4.85bpw, calibrated with images as well as text. Costs +0.53 GB over that build and buys two things: full prefill speed with no workaround, and a vision tower that was actually exercised during calibration.

No 3-bit anywhere: oMLX's qwen35_prefill kernels expose q2/q4/q5/q6/q8 but not q3, so 3-bit in the MLP silently drops prefill to a slow path.

Why group size 64

oMLX routes 4-bit MLP matmuls to a native kernel above 2048 tokens, and that kernel's speed comes from the NAX tensor-unit path — which is gated to group_size == 64. At gs128 it demotes to a slower path, costing ~1.8× prefill on M5 hardware unless you disable the routing (jundot/omlx#2657).

Measured at pp 4096 / tg 128, no environment variables, same machine:

So gs64 gets essentially full prefill speed with no workaround and no local patching, which is the practical argument for the extra 0.53 GB.

Speed

M5 Max, oMLX, no environment variables. Two separate options are on: native MTP (the checkpoint's built-in multi-token-prediction heads, accelerating decode) and SpecPrefill with a small draft model (accelerating prefill above 8K tokens).

A note on the SpecPrefill drafter, because the obvious choice is not the fast one. SpecPrefill feeds the target's token ids straight to the drafter, so a drafter from the same tokenizer family (vocab_size 248,320, e.g. Qwen/Qwen3.5-0.8B) is the only one whose importance scores describe the actual prompt — a smaller-vocab drafter gets zero vectors for every out-of-range id rather than an error, and shares only 0.19% of id-to-token meanings with this model.

Measured, though, a 4-bit Qwen2.5-0.5B drafter is faster and much lighter at long context — 1467 vs 876 ppTPS and 33.1 vs 53.1 GB peak at 200K — because the drafter must prefill every token to score it, and a Qwen3.5-family drafter carries head_dim 256, GDN state and a 248k-row embedding.

Both figures above are throughput only. Whether the tokenizer mismatch costs long-context accuracy is not measured here, and the quality benchmarks in this card all sit below SpecPrefill's 8192-token trigger, so they do not answer it either. Test retrieval at depth on your own workload before committing.

Two separate accelerators are active, and they show up in different columns. Native MTP drives the decode side (tgTPS). SpecPrefill drives the prefill side: a small draft model scores token importance and only the top ~20% are prefilled on the target. It engages above 8192 tokens, which is exactly where ppTPS steps from ~810 to ~2500 and TTFT falls from 10.1 s at 8K to 6.6 s at 16K. Those tail rows are still cold prefills — they are sparse, not cached — so they are legitimately faster, just not measuring the same work as the ≤8K rows.

The tail also shows the memory story: the full 200K window costs 33.1 GB and still decodes at 30 tok/s.

Batched, same machine:

Calibration

352 prompts × 1024 tokens, 234,477 real tokens, rendered through Qwen's own chat template with a 50/50 think/nothink split. Unlike the 4.85bpw build, this one includes images, so the vision tower actually executed during calibration rather than having its bits assigned by guesswork:

Sequential AWQ over the dense MLP: each layer is calibrated on activations from the already-quantized layers above it. Everything outside the MLP is RTN at the widths above.

MTP

The MTP head ships inside the checkpoint (31 tensors under mtp.*) and is quantized: 8-bit attention, 6-bit fc, 4-bit MLP. Turn it on with oMLX's mtp_enabled, which uses those heads to draft and verify during decode.

Fixed 2026-08-16 — re-download if you pulled this before that date. The head shipped in raw-HF norm convention while the backbone was already converted to MLX's, so mtp.layers.0.input_layernorm averaged 0.0361 instead of ~1.036 and two norms were negative. Drafts stopped matching the target and acceptance collapsed. On a loader with no compensation this made MTP slower than no speculation at all (22.4 vs 24.6 tok/s); repaired, the same setup runs 43.7 tok/s. Inside oMLX the damage was partly masked — its norm_repair shifts any head norm averaging below 0.5, which caught 3 of the 7 and left q_norm (0.78), k_norm (0.79) and mtp.norm (1.25) raw; repairing those three measured +9% decode (median 53.3 vs 48.8 tok/s, pp 4096 / tg 128, three runs each, no overlap between the groups).

Only model-00005-of-00005.safetensors changed. Benchmarks below are unaffected: they were run with MTP off, and MTP is rejection-verified, so a degraded head costs acceptance rate but never output correctness.

Quantizing the head is safe — drafts are rejection-verified by the target model, so head error costs acceptance rate, never output correctness. Leaving it at bf16 inside a quantized model measurably halves throughput.

MTP is a distinct feature from speculative decoding with an external draft model, and the two accelerate different phases. They compose: the speed table above was measured with native MTP on the decode side and SpecPrefill driving prefill from a Qwen2.5-0.5B-Instruct draft model, which is what produces the ~3× prefill step above 8K.

Quality

Same questions, same harness, against the 4.85bpw sibling:

The totals are identical. Three questions lost on HumanEval, three regained across GSM8K and MMLU — that is what noise looks like at this sample size, not a capability difference, and neither per-benchmark gap is significant on its own. Both builds sit four questions behind bf16.

So the extra 0.53 GB does not buy accuracy. It buys the prefill speed above without an oMLX patch, and a vision tower that was calibrated on real images rather than assigned its bit width by guesswork. Pick on that basis.

One caveat on attribution: this build changes two variables at once relative to the 4.85bpw one — group size and calibration data. Since the aggregate result is a tie, neither variable moved the needle enough to need separating.

Requires oMLX

The checkpoint keeps its mtp.* weights so the MTP head can bind. Stock mlx_vlm sees those keys, flips its should_shift_norm_weights heuristic, and applies a +1.0 offset to layernorms that already have it — the model still loads and still produces fluent text, but the text is wrong. oMLX patches sanitize to gate that on conv1d layout instead, which is correct here.

Do not load this outside oMLX.

Sampling

The chat template takes reasoning_effort of xhigh (default), medium or low, and raises on anything else. It also accepts enable_thinking and preserve_thinking. Pass via chat_template_kwargs.

Caveats

• 27 vision linear_fc2 modules stay at bf16: their input dim (4304) is not divisible by any supported group size.
• Built with mlx-compress.