Qwen3.6-27B-Mixed-4bit

Qwen3.6-27B-Mixed-Bit-MLX

TLDR: 4-bit MLP crush delivers 89% speedup with zero intelligence loss.

Compressed MLP, uncompressed attention, vision, and MTP (Apple Silicon Optimized mixed quantization).

Three quantizations are provided:

• Qwen3.6-27B-Mixed-8bit — Conservative, 36 GB, 14 tok/s
• Qwen3.6-27B-Mixed-6bit — Balanced, 32 GB, 16 tok/s
• Qwen3.6-27B-Mixed-4bit — Aggressive, 28 GB, 18 tok/s

All three preserve full BF16 precision in attention, vision, and multi-token prediction layers. The 4-bit version matches the 8-bit on coding benchmarks. The 8-bit is more stable for long context (100-200k token) agentic work.

Source model: https://huggingface.co/Qwen/Qwen3.6-27B

---

(1) Introduction

The Objective: Get more throughput out of Qwen3.6-27B on Apple Silicon without sacrificing the reasoning and context-window recall that make it useful for agentic work and coding.

The Problem: BF16 gives you the full brain but chugs at ~9 tokens/sec on M5 Max due to memory bandwidth. Uniform quantizations are faster but degrade the hybrid attention mechanism and vision tower. The compressed attention damages agentic ability.

The Solution: Mixed Quantization. A surgical approach that compresses only the SwiGLU MLP layers — which make up 66% of the model's mass — to 8-bit, 6-bit, or 4-bit, while leaving every attention projection, the vision tower, the MTP head, embeddings, and norm layers at full BF16 precision.

The result: 17 tokens/sec at 4-bit with nearly identical benchmark scores, or 14–16 tok/s at 8/6-bit if you want to be conservative.

Resulting sizes:

---

(2) Architecture

Qwen3.6-27B is a dense hybrid-attention model. It has 64 transformer layers following a repeating pattern:

[linear_attention, linear_attention, linear_attention, full_attention] × 16 = 64 layers

The linear attention layers use SSM-style state-space projections (in_proj_qkv/z, conv1d, A_log, dt_bias) for efficient long-context streaming across a 262K token window. Every fourth layer uses full softmax attention for deep reasoning.

Additionally, the model includes:

• A 27-layer vision tower for image/video comprehension
• A 1-layer MTP (Multi-Token Prediction) head for speculative decoding
• SwiGLU MLPs in every layer for feature transformation

Identifying the Weight

Out of 27B parameters, the MLP projections (gate_proj, up_proj, down_proj) make up 66% of the model's mass — 32 GB of the 49 GB BF16 footprint. These SwiGLU layers are the "knowledge lookup" / feature transformation matrices and are extremely tolerant of quantization.

The Glass Cannons: Layers Protected at BF16

The Crush Zone

---

(3) The Quantization Script

The script uses MLX's native nn.quantize() with a class_predicate filter that targets only the three SwiGLU projection matrices inside language_model.layers.N.mlp. Everything else passes through untouched.

import mlx.core as mx
import mlx.nn as nn
from mlx_vlm.utils import load
import json, os, gc, argparse

parser = argparse.ArgumentParser()
parser.add_argument("--bits", type=int, default=8, choices=[8, 6, 4])
parser.add_argument("--group-size", type=int, default=64)
args = parser.parse_args()

model_path = "/path/to/Qwen3.6-27B"
output_path = f"/path/to/Qwen3.6-27B-Mixed-{args.bits}bit"

# Lazy load — no eager evaluation
model, processor = load(model_path, lazy=True)

def mlp_only_filter(path, module=None):
    """Target ONLY SwiGLU MLP projections. Protect everything else."""
    if "visual" in path:
        return False
    if path.startswith("mtp") or ".mtp." in path:
        return False
    if "embed_tokens" in path or "lm_head" in path:
        return False
    if "layernorm" in path or path.endswith(".norm"):
        return False
    if ".mlp." not in path:
        return False
    leaf = path.split('.')[-1]
    return leaf in ("gate_proj", "up_proj", "down_proj")

# Single-pass quantization — only MLP gets touched
nn.quantize(model, group_size=args.group_size, bits=args.bits,
            class_predicate=mlp_only_filter)

# Save config and processor
os.makedirs(output_path, exist_ok=True)
if hasattr(processor, "save_pretrained"):
    processor.save_pretrained(output_path)

with open(os.path.join(model_path, "config.json")) as f:
    config = json.load(f)
config["quantization"] = {
    "group_size": args.group_size,
    "bits": args.bits,
    "type": "mixed",
    "description": f"MLP-only {args.bits}-bit. Attention, vision, MTP, embeddings, norms remain BF16."
}
with open(os.path.join(output_path, "config.json"), "w") as f:
    json.dump(config, f, indent=4)

# Flatten, destroy model tree, shard to SSD
flat_weights = dict(flatten_parameters(model.parameters()))
del model, processor
gc.collect()
mx.clear_cache()

# Sequential eval + shard (5 GB chunks)
current_shard, shard_size, shard_idx = {}, 0, 1
for name, tensor in flat_weights.items():
    if not isinstance(tensor, mx.array):
        continue
    mx.eval(tensor)
    current_shard[name] = tensor
    shard_size += tensor.nbytes
    if shard_size >= 5 * 1024**3:
        mx.save_safetensors(
            f"{output_path}/model-{shard_idx:05d}.safetensors", current_shard)
        current_shard, shard_size, shard_idx = {}, 0, shard_idx + 1
        gc.collect(); mx.clear_cache()
if current_shard:
    mx.save_safetensors(
        f"{output_path}/model-{shard_idx:05d}.safetensors", current_shard)

Full script: mixed_quantize_qwen3_6_27b.py in the quantization-scripts repository.

---

(4) Bypassing Framework Bugs

Critical deployment information

The mlx_vlm sanitize_weights Panic

Issue: The mlx_vlm library searches for HuggingFace-style tensor key names during its sanitize_weights phase. Because this model is natively quantized in MLX format, the key structure differs, causing a crash or fallback to LLM-only mode.

Fix: Locate utils.py in your mlx_vlm installation and add return weights as the first line of the sanitize_weights function to bypass the check entirely as follows:

Update: No hacks required to load these models as VLMs.

Note on config.json

The config.json declares a single "bits" value, but MLX reads per-layer quantization metadata dynamically from the safetensors tensor headers. The actual mixed-precision structure — which layers are quantized and which remain BF16 — lives in the safetensors files, not the config. Tooling that trusts only config.json will be confused; this is expected behavior for native MLX mixed quantization.

---

(5) Real-World Performance on Apple Silicon

Hardware: M5 Max, 128 GB Unified Memory.

Baseline: Qwen3.6-27B BF16 (~49 GB)

Coding Benchmarks

**Model: Qwen3.6-27B (BF16)**

Benchmark         Accuracy   Correct   Total   Time(s)
------------------------------------------------------
HUMANEVAL            92.1%       151     164    1602.4
MBPP                 85.5%       171     200    1953.8
LIVECODEBENCH        60.0%        60     100    8920.9


**Model: Qwen3.6-27B-Mixed-8bit**

Benchmark         Accuracy   Correct   Total   Time(s)
------------------------------------------------------
HUMANEVAL            91.5%       150     164    1049.9
MBPP                 86.5%       173     200    1161.3
LIVECODEBENCH        62.0%        62     100    5600.6


**Model: Qwen3.6-27B-Mixed-6bit**

Benchmark         Accuracy   Correct   Total   Time(s)
------------------------------------------------------
HUMANEVAL            92.1%       151     164    1029.0
MBPP                 86.0%       172     200    1231.5
LIVECODEBENCH        57.0%        57     100    5152.3


**Model: Qwen3.6-27B-Mixed-4bit** 

Benchmark         Accuracy   Correct   Total   Time(s)
------------------------------------------------------
HUMANEVAL            90.8%       149     164     893.1
MBPP                 86.5%       173     200    1074.2
LIVECODEBENCH        62.0%        62     100    3290.5

---

(6) Conclusion

Mixed MLX quantization of Qwen3.6-27B that compresses only the MLP layers — while preserving all attention, vision, and MTP layers at BF16 — delivers up to an 89% speedup with no statistically significant intelligence loss.

The 4-bit version is the sweet spot: at 28 GB and 17 tok/s, it matches the 8-bit on MBPP and LiveCodeBench (62.0% both), and trails BF16 by only 1.3% on HumaneVal — well within benchmark variance. It completes LiveCodeBench in 33 seconds per problem vs. 89 seconds for BF16, making it dramatically more practical for real-time agentic coding workflows.

The hybrid attention architecture (linear SSM + full softmax) is the key: by protecting both attention types in BF16, the model retains its full 262K context-window recall and reasoning capability. The MLP layers — which are just feature transformation matrices — absorb even 4-bit quantization with no measurable penalty on coding tasks.

Notes

• Fixed seed sampling was used on all benchmarks.
• MBPP and LiveCodeBench were run on subsamples (200/500 and 100/1055 respectively).
• Test times include multitasking overhead and are not strictly comparable.
• Vision capabilities were preserved but not formally benchmarked in this pass.