Qwen3.8-27B-MLX-OptiQ-4bit

Qwen3.8-27B-MLX-OptiQ-4bit

A mixed-precision quant of Qwen/Qwen3.8-27B for Apple Silicon, built with mlx-optiq at 5.63 bits per weight. 4bit names the dominant precision, not the average.

Tip: the 5bit is smaller and measures better overall — German prose is the one span where this build stays ahead.

Highlights

• 🖼️ Image input on all three stacks: optiq serve, mlx-vlm, vllm-mlx. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
• 📏 Measured weights: per-layer {4, 8} bit from exact KL vs the bf16 base. No structural rules — every tensor's bit-width is bought by measured sensitivity. Results in the table below.
• 🧠 Measured KV cache: per-layer bit-widths, zero tool-call flips. Ships as optiq/kv_config.json — details below.
• 🔧 Hardened chat template, adopted from unsloth. Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).
• ⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search.

How it compares

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Everything at once — image input, measured KV cache, MTP speculation:

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-4bit --mtp \
  --kv-config "$(hf download p4ik/Qwen3.8-27B-MLX-OptiQ-4bit optiq/kv_config.json)"

Image input with mlx-vlm:

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")
config = load_config("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")

prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"]))

Text-only with mlx-lm:

from mlx_lm import load, generate

model, tokenizer = load("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")
messages = [{"role": "user", "content": "Write a merge sort in Python."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt))

Quantization

optiq convert Qwen/Qwen3.8-27B \
    --method optiq --reference bf16 \
    --target-bpw 5.0 --candidate-bits 4,8 \
    --n-calibration 67 --calibration-mix optiq_erweitert.jsonl \
    --skip-baselines

This build replaced an earlier static (rule-based) allocation: a paired corpus comparison put the two at parity, and the measured allocation ships because it is documented evidence rather than convention. The per-layer KL table is in optiq/sensitivity.json, the final assignment in optiq/metadata.json.

KV cache

8-bit on layers 51, 55, 59, 63; 4-bit on the other twelve (5.00 bits average; only the 16 full_attention layers carry a KV cache — the 48 linear_attention layers hold a fixed-size Gated DeltaNet state).

Measured with paired next-token NLL over 196k tokens (agentic transcripts with on-policy thinking and tool calls, German prose, WikiText), each layer probed individually. At this allocation, tool calls, thinking and English text are statistically indistinguishable from a full 8-bit cache; German prose retains ~+0.4% perplexity. A flip check over near-certain tokens found zero flips inside tool-call spans — the 4-bit cache does not break tool-call syntax.

Files

Sampling

From the base model card, unchanged:

Thinking mode is on by default and can be disabled per request. Context is 262,144 tokens natively, extensible to 1M with YaRN.