Qwen3.8-27B-MTPLX-4bit

Qwen3.8-27B-MTPLX-4bit

MTP weights kept. This build does not strip the model's multi-token-prediction head — it is preserved at full bf16 precision (mtp_policy: keep_bf16) alongside the 4-bit quantized body, so native speculative decoding still works.

A 4-bit MLX build of Qwen/Qwen3.8-27B for Apple Silicon, produced with mtplx forge. Unlike a standard mlx_lm.convert quantization, this build preserves the model's native multi-token-prediction (MTP) head instead of discarding it, so it runs with real speculative decoding on mtplx — not just a quantized weight dump.

Why this build exists

Qwen3.8-27B is a hybrid architecture (Gated DeltaNet linear attention interleaved with Gated Attention, 16 groups of 3:1). Standard MLX conversion paths (mlx_lm.convert, and MLX inference engines built around dense-attention assumptions) silently strip the model's 15 MTP tensors during conversion — the model still loads and generates, but you lose the native speculative-decoding speedup entirely, often without any error or warning.

This build's recipe uses mtp_policy: keep_bf16, which retains the MTP block in full precision alongside the quantized body, and validates it with mtplx's tune verification suite (long-code-uncapped, 2048 max tokens) before shipping.

Quantization recipe

Group size 64 was chosen over the tighter g32 default because the target hardware (Apple Silicon with 64GB+ unified memory) has headroom to trade a small amount of table overhead for faster dequantization, with no measurable quality cost at this bit depth.

Requirements

• Apple Silicon Mac (M-series)
• mtplx — required to use the native MTP speculative-decode path. Standard mlx-lm can load this model's body weights but will not use the MTP head.

Usage

mtplx quickstart --model johninthepool/Qwen3.8-27B-MTPLX-4bit --port 8020 \
  --reasoning off --paged-kv-quantization q8

--reasoning off is recommended for agentic/tool-calling workloads — leaving reasoning on causes this model to spend a large number of tokens per turn on hidden thinking before it produces usable output.

--paged-kv-quantization q8 quantizes the KV cache to 8-bit, giving substantially larger usable context at negligible quality cost. Qwen3.8-27B natively supports up to 262,144 tokens of context (extendable to ~1M with YaRN).

What you get vs. what you give up

You get: native speculative decoding via the preserved MTP head, full 4-bit body compression (~17GB on disk), long-context serving via paged KV quantization.

You give up relative to the 8-bit sibling: a small amount of body-weight precision. If output quality matters more than footprint/throughput for your use case, see johninthepool/Qwen3.8-27B-MTPLX-8bit.

Provenance

Built from the original Qwen/Qwen3.8-27B weights with no fine-tuning, distillation, or calibration pass applied — this is a direct affine (round-to-nearest) quantization of the release weights, with the MTP head kept unquantized. No behavioral changes beyond quantization noise are expected relative to the source model.