Qwen3.8-27B-MLX-4bit

Qwen3.8-27B-MLX-4bit

A uniform 4-bit MLX quantization (group size 64) of Qwen/Qwen3.8-27B for Apple Silicon — as a full package: bf16 vision tower, MTP speculative-decoding head, processor configs, hardened chat template. Every layer at 4 bits, no mixed precision.

Tip: the smallest full package of this base. For quality per byte take the OptiQ-5bit — a third of the flip rate at +3.6 GiB.

Highlights

• 🖼️ Image input on all three stacks: optiq serve, mlx-vlm, vllm-mlx. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
• ⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search.
• 🔧 Hardened chat template, adopted from unsloth. Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).

How it compares

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Image input, uniform 4-bit KV cache, MTP speculation — one line:

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-4bit --mtp --kv-bits 4

--kv-bits 4 (group size 64) matches the cache to the all-4-bit footprint — the smallest total. For measured per-layer KV, take the OptiQ repos. Text-only use works with plain mlx-lm; image input also runs under mlx-vlm and vllm-mlx.

Files

No measured KV config and no sensitivity table — those are products of the measured OptiQ builds.

Sampling

From the base model card, unchanged: temperature 1.0 / top_p 0.95 (thinking), 0.7 / 0.8 (instruct).