Qwen3.8-27B-Uncensored

Qwen3.8-27B-Uncensored — MLX 5.0 bpw (gs64, NAX-first)

A mixed-precision MLX affine quantization of orcarouter/Qwen3.8-27B-Uncensored (27.8 B dense, 64 layers, hybrid GatedDeltaNet + full attention, 256 K context, MTP head, vision tower), built to replicate the recipe of True2456/Qwen3.8-27B-AWQ-5.0bpw and tuned for oMLX on M5 (NAX tensor unit).

17.38 GB · 5.006 bpw · 3.2× smaller than bf16

Credits

• True2456 — the recipe this build replicates: the per-module bit map and its rationale, the group-size-64 argument for oMLX's NAX prefill path, the no-3-bit constraint (oMLX's qwen35_prefill kernels ship q4/q5/q6/q8 only), and the MTP and norm-convention findings. Built on their mlx-compress.
• OrcaRouter — the bf16 source weights, Qwen3.8-27B-Uncensored (abliterated fine-tune, revision 11a200330368fd04fa7b2f0b1e7f20bd819cf621).
• Qwen — the original Qwen/Qwen3.8-27B.

Per-module widths

No 3-bit anywhere, and gs64 on everything that can route — gs128 is refused by oMLX's NAX path when the tensor unit is available.

How this differs from the recipe it replicates

1. SLQ-style GPTQ on the 8-bit groups instead of plain RTN: one-shot block GPTQ, asymmetric, rel_damp 1e-2, Hessians from real prefill activations of this checkpoint. Measured on the stored bytes: 48/48 attention modules improved, median −32 % H-weighted output error, +31.2 % aggregate. It re-rounds on the grid the export already chose, so only packed codes change.
2. Vision qkv/proj at gs64 rather than gs128 (+4 MB), because oMLX refuses gs128 on the NAX path — gs64 is what actually reaches the kernel.
3. No AWQ on the MLP. It was implemented and measured: +32.8 % in-sample, but −36.6 % on held-out prompts, i.e. the per-channel scales and per-group clip ranges fitted the calibration activations rather than the model. Reverted rather than shipped. The MLP carries the uniform-grid range search instead.

Running it

Serve through oMLX. Stock mlx_vlm.load fails on the 31 MTP tensors (Received 31 parameters not in model) and its should_shift_norm_weights heuristic would double-shift norms that are already in MLX convention here. oMLX gates that heuristic on conv1d layout instead. For a trunk-only load, pass strict=False.

• Enable native MTP: the head ships inside the checkpoint; drafts are rejection-verified by the trunk, so head error costs acceptance rate, never output correctness.
• The chat template takes reasoning_effort of xhigh (default), medium or low and raises on anything else, plus enable_thinking and preserve_thinking — pass via chat_template_kwargs.
• For speculative prefill, use a drafter from the same tokenizer family (248,320 vocab), e.g. Qwen/Qwen3.5-0.8B or Qwen/Qwen3.5-2B; a smaller-vocab drafter scores out-of-range ids as zeros.

Measured quality

Per-tensor, dequantized stored bytes vs the bf16 source (rel-L2 / cosine):

Vision 8-bit and the MTP attention keep the range search rather than GPTQ: no image calibration data was available, so their Hessians could not be captured honestly.

License

Apache-2.0, inherited from the source checkpoint and the Qwen base model. The quantization changes weights only; behaviour and any limitations of Qwen3.8-27B-Uncensored carry over unchanged.