Qwen3.6-35B-A3B-MLX-3bit-mixed_3_8

Qwen3.6-35B-A3B — MLX 3-bit mixed_3_8 (MoE-protected, with vision)

A mixed-precision MLX conversion of Qwen/Qwen3.6-35B-A3B, quantized with a custom mixed_3_8 recipe that protects MoE router gates and preserves the full vision tower at bf16.

Effective size: ~3.5 bpw language + bf16 vision, ~19 GB on disk. Runs on any Apple Silicon Mac with 16 GB+ unified memory.

---

What is this?

Qwen3.6-35B-A3B is a Mixture-of-Experts model from Alibaba's Qwen team. Despite the "35B" name, only 3 billion parameters are active per token — it has 256 experts but routes through only 8 at a time. That makes it far more efficient to run than a dense model of equivalent size.

This repo contains the full multimodal model — language tower + vision tower. It handles text, images, and video.

---

Quantization method

Why mixed_3_8?

mixed_3_8 is a per-layer mixed-precision recipe — some layers get 3-bit, others get 8-bit. The bit allocation follows the same logic as llama.cpp's Q4_K_M:

• First and last 1/8 of layers → 8-bit
• Every 3rd layer in the middle → 8-bit
• v_proj, down_proj (when layer qualifies) → 8-bit
• lm_head → 8-bit
• Everything else → 3-bit

3-bit keeps the bulk of the model small; 8-bit preserves the layers that matter most. The result is near-lossless quality on sensitive layers while still meaningfully shrinking the model.

MoE router protection

The built-in mixed_3_8 recipe doesn't know about MoE architectures. For this conversion, a custom predicate forces all router.gate and router.linear weights to 8-bit. Without this, low-bit routing decisions cause silent quality degradation — tokens get sent to the wrong experts and the output drifts.

Vision tower

The vision encoder is left at bf16 (unquantized). ViTs are very sensitive to low-bit quantization, and the vision tower is small enough (~1.5 GB) that leaving it at full precision is the safe default.

---

Quick start

pip install mlx-vlm

from mlx_vlm import load, generate

model, processor = load("hermitdave/Qwen3.6-35B-A3B-MLX-3bit-mixed_3_8")

messages = [{"role": "user", "content": "Describe this image.", "image": "path/to/image.jpg"}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

result = generate(model, processor, prompt=prompt, max_tokens=200)
print(result.text)

Or via CLI:

python -m mlx_vlm.generate --model hermitdave/Qwen3.6-35B-A3B-MLX-3bit-mixed_3_8 --max-tokens 100 --prompt "Describe this image." --image path/to/image.jpg

---

Model properties

---

Hardware

• Minimum: 16 GB unified memory (M1/M2/M3/M4)
• Recommended: 32 GB+ for comfortable context headroom
• The MoE architecture means CPU offloading is less painful than on dense models — only active experts are computed per token.

---

Known limitations

• Thinking mode. The chat template defaults to reasoning enabled (enable_thinking: true). To disable, pass chat_template_kwargs={"enable_thinking": false} at inference time.
• Hybrid attention profile. Prefill is slower than decode due to the recurrent linear-attention layers. This is expected behavior, not a bug.

---

How this was made

python -m mlx_vlm.convert \
  --hf-path Qwen/Qwen3.6-35B-A3B \
  --mlx-path ./Qwen3.6-35B-A3B-MLX-3bit-mixed_3_8 \
  -q --quant-predicate mixed_3_8 \
  --trust-remote-code

The router-protected predicate is available as a Python snippet — load the model with lazy=True, build a predicate that maps router.gate paths to 8-bit, skips the vision tower, and applies standard mixed_3_8 to everything else, then pass it to convert().

---

Acknowledgments

• Qwen team at Alibaba Cloud for the base model.
• MLX team at Apple for the framework.
• mlx-vlm for the conversion tools.
• Nous Research for Hermes Agent, which handled the conversion and upload.

---

License

Apache 2.0. Inherited from the upstream model.