Qwen3.6-35B-A3B-MLX-4.8bit

Qwen3.6-35B-A3B optimized for MLX.

• 4-bit baseline with important layers at 8-bit and BF16.
• This quant does not support image input.

I ended up selecting two winners from my trials. This is the speed+ version, and here's the quality+ version.

Usage

# Start server at http://localhost:8080/v1/chat/completions
uvx --from mlx-lm mlx_lm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Qwen3.6-35B-A3B-MLX-4.8bit

Benchmarks

I've moved over to using speed + KL divergence as my primary optimization metrics. Hellaswag, PIQA, Winogrande, and perplexity are kept as sanity checks, though these require high sample sizes to get usable signal.

Tested on a Mac Studio M3 Ultra with:

mlx_lm.convert --hf-path Qwen/Qwen3.6-35B-A3B --mlx-path ./mlx && mlx_lm.kld --baseline-model ./mlx
mlx_lm.perplexity --sequence-length 512 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 500
mlx_lm.evaluate --tasks piqa --seed 123 --num-shots 0 --limit 500
mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 500

mlx_lm.kld is still an open PR.

Methodology

Quantized with a mlx-lm fork, drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. MLX quantization options differ than llama.cpp, but the principles are the same:

• Sensitive layers like MoE routing, attention, and output embeddings get higher precision
• More tolerant layers like MoE experts get lower precision