Qwen3.5-35B-A3B-UD-Q8_K_XL-mlx

Qwen3.5-35B-A3B-UD-Q8_K_XL — MLX

MLX-compatible 8-bit quantized weights for Qwen3.5-35B-A3B, ported from the Unsloth Q8_K_XL GGUF.

• Quantization: 8-bit, group_size=64, bfloat16 scales/norms
• Speed: ~81 tok/s generation, ~195 tok/s prompt processing (M4 Max, 128GB)
• Peak memory: ~37 GB

Why This Model?

We ported from Unsloth's Q8_K_XL GGUF rather than using the existing mlx-community 8-bit conversion because Unsloth's quantization produces superior generation quality. The Q8_K_XL format selectively preserves higher precision on sensitivity-critical layers (SSM projections at fp16), resulting in better output fidelity than uniform quantization from the HuggingFace source weights. This port brings that quality advantage to MLX while matching or exceeding the speed of the standard mlx-community conversion.

Porting Process

1. GGUF → MLX Conversion

Converted from Qwen3.5-35B-A3B-UD-Q8_K_XL.gguf using a custom converter that maps GGUF tensor names to the MLX qwen3_5 architecture. The Q8_K_XL format intentionally keeps SSM projection layers at full precision (fp16) for accuracy, while MoE expert weights use Q8_K quantization (block_size=256).

2. Value-Head Fix

The initial conversion produced garbage output due to incorrect weight mapping for the GatedDeltaNet (linear attention) value projection layers. Fixed the converter to correctly map in_proj_b (beta) and in_proj_a (alpha/dt) projections from the GGUF layout.

3. Post-Conversion Re-Quantization

The raw GGUF→MLX conversion ran at ~4 tok/s — roughly 20x slower than expected. Three issues were identified and fixed via a post-hoc re-quantization pipeline:

Problem 1: fp16 SSM layers. The Q8_K_XL format keeps SSM projection layers (in_proj_qkv, in_proj_z, out_proj, etc.) at fp16 for precision. However, MLX's batch=1 fp16 matmul uses a generic Metal GEMM kernel, while its quantized path uses a fused dequantize-matvec kernel that is 20–25x faster for autoregressive decode.

Problem 2: group_size=32 MoE weights. GGUF Q8_K uses block_size=256, which maps to group_size=32 in MLX's quantization scheme (256 elements / 8 bits = 32 scales). MLX's Metal quantized kernel is ~8x slower for group_size=32 vs group_size=64 (the setting used by mlx-community/Qwen3.5-35B-A3B-8bit).

Problem 3: dtype mismatches. GGUF stores norm weights as float32 and scales as float16. The official mlx-community model uses bfloat16 throughout. Apple Silicon has native bfloat16 hardware paths; mixed float16/float32 introduces unnecessary dtype conversion overhead.

Fix: A single re-quantization script performs:

1. Load the converted model
2. Dequantize all quantized layers back to full precision
3. Cast all float16/float32 weights to bfloat16 (703 tensors)
4. Re-quantize all linear layers uniformly to 8-bit, group_size=64
5. Save the final model

This brought generation speed from ~4 tok/s → 81 tok/s (exceeding the official mlx-community 8-bit model at 72.5 tok/s on the same hardware).

Usage

from mlx_lm import load, generate

model, tokenizer = load("RyanSB/Qwen3.5-35B-A3B-UD-Q8_K_XL-mlx")
response = generate(model, tokenizer, prompt="Hello", max_tokens=100)

Or via mlx_lm.server:

python -m mlx_lm server \
  --model RyanSB/Qwen3.5-35B-A3B-UD-Q8_K_XL-mlx \
  --chat-template-args '{"enable_thinking":false}'

Benchmarks

Hardware: Apple M4 Max, 128 GB unified memory.