Qwen3.6-27B-MLX-mixed-4bit

Qwen3.6-27B-MLX-mixed-4bit

A sensitivity-fenced, mixed-precision 4-bit quantization of Qwen/Qwen3.6-27B for Apple Silicon (MLX). Most of the network runs at 4-bit for speed and size, while the handful of weights that are genuinely sensitive to low-bit quantization are kept at 6/8-bit or BF16.

• 5.36 bits/weight, ~17 GB on disk
• ~30 tok/s decode, ~18 GB peak memory on an Apple M5 Max
• Weight-only affine quant (activations stay FP16) → no activation-outlier failure modes
• Beats a naïve uniform-4bit build on every fidelity metric (see below)

TL;DR on choosing a build: if you have the memory, the 6-bit build is effectively lossless. This mixed-4bit build is for when you want maximum decode speed and the smallest footprint and can accept a measurable (~5% perplexity) quality cost.

Why mixed precision?

Qwen3.6-27B is a hybrid model: 48 of its 64 layers use Gated-DeltaNet linear attention (a recurrence that carries a state across the sequence) and 16 use full softmax attention. We profiled the int4 reconstruction error of all 496 linear tensors. The finding: int4 error is remarkably uniform (~0.091 relative) everywhere except a small set of tensors — the DeltaNet decay/control gates (in_proj_a/in_proj_b) and the GQA k/v projections. Those feed the recurrent state, where a small error compounds over the sequence. So we quantize the bulk to 4-bit and fence exactly those tensors at higher precision — at negligible size cost, since they are tiny.

Precision map

Resulting distribution by parameter mass: ~68% 4-bit, ~22% 6-bit, ~10% 8-bit, <1% BF16.

Measured quality (vs BF16)

Perplexity, KL-divergence, and top-1 agreement against the BF16 model on a fixed 650-token multi-domain passage. This is an indicative smoke test, not a full benchmark suite — treat it as a relative comparison between builds, not an absolute quality guarantee. Hard-reasoning/long-context degradation is typically larger than perplexity suggests.

¹ within noise — 6-bit is statistically indistinguishable from BF16 on this sample.

The fencing cuts KL-divergence by 38% vs uniform-4bit for +0.86 bpw. Note that because the MLP (61% of params) is 4-bit in both, weight-only fencing cannot close the fundamental 4-bit "cliff" (+5% ppl); doing that requires distillation-based quant (DWQ) or rotations.

Usage

pip install mlx-lm
mlx_lm.generate --model EigenLabs/Qwen3.6-27B-MLX-mixed-4bit \
  --prompt "Explain gated delta networks in two sentences." --max-tokens 256

from mlx_lm import load, generate
model, tokenizer = load("EigenLabs/Qwen3.6-27B-MLX-mixed-4bit")
messages = [{"role": "user", "content": "What is 17*23?"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

Qwen3.6 is a reasoning model — the chat template opens a <think> block, so responses begin with a chain of thought.

Requirements & notes

• Apple Silicon with MLX; ~20 GB free unified memory (weights + KV). Comfortable on 32 GB+, ideal on 48 GB+ for long context.
• Text-only. The MLX qwen3_5 implementation does not load the model's MTP head (so no built-in speculative decoding) or its vision tower (use mlx-vlm separately for images).
• The hybrid Gated-DeltaNet layers run on the Metal GPU (no CPU fallback observed).

How it was made

Quantized with mlx_lm.convert using a custom mixed-precision predicate (asymmetric affine, group_size=64):

def predicate(path, module):
    if "in_proj_a" in path or "in_proj_b" in path:      return False          # keep BF16
    if "embed_tokens" in path or "lm_head" in path:     return bits(8)
    if "k_proj" in path or "v_proj" in path:            return bits(8)
    if "in_proj_z" in path or "out_proj" in path:       return bits(6)
    if "q_proj" in path or "o_proj" in path:            return bits(6)
    if "down_proj" in path and edge_layer(path):        return bits(6)
    return bits(4)                                                            # MLP bulk, in_proj_qkv

Tokenizer, chat template, and generation config are copied verbatim from the base model.

License & attribution

Apache 2.0, inherited from Qwen/Qwen3.6-27B (© Alibaba / Qwen team). This is a quantized derivative produced by EigenLabs. Please cite the original Qwen3.6 model.