Qwen3.6-35B-A3B-oQ8e-fp16-mtp

Qwen3.6-35B-A3B-oQ8e-fp16-mtp

An oQ mixed-precision MLX quantization of Qwen/Qwen3.6-35B-A3B, built for Apple Silicon.

Highest-precision of the three. Heaviest weights, so it leaves the least room for KV cache.

Quantization

Mixed precision, 8-bit affine base at group size 64, with 265 per-module overrides at 8-bit/group size 128.

Bit-width assignment is imatrix-guided (oqe_code_multilingual, 128 samples x 512 tokens) — 481 of 523 collected entries applied. Layers left unquantized are kept at fp16, which is what the fp16 in the repo name refers to. The mtp suffix means the checkpoint retains its multi-token-prediction head (1 MTP layer, shared embeddings), so MTP speculative decoding works without a separate draft model.

• Format: MLX safetensors, 39.5 GB across 2,052 tensors
• Mode: affine
• Calibration report: oq_imatrix_report.json is included in this repo

Architecture

This is a hybrid-attention model: only the 10 full-attention layers grow a KV cache with sequence length. The remaining 30 linear-attention layers hold a fixed-size recurrent state.

Measured performance

Single-stream, measured on an Apple M1 Max / 64 GB running oMLX. 300-token generations (median of 3) and a 7.5k-token uncached prefill.

These are throughput figures on one machine, not a quality benchmark, and they will vary with hardware, context length and sampler settings. The first request after a model load reads low — that is warm-up, not steady state.

Memory planning

At fp16 the KV cache costs roughly 21 KB per token (10 full-attention layers x 2 KV heads x 256 head dim x 2 for K/V x 2 bytes).

Budget for weights plus KV cache plus runtime overhead. If the cache does not fit, quantizing it (for example TurboQuant 8-bit) roughly halves these figures at no measurable throughput cost.

Usage

Served through oMLX over an OpenAI-compatible API:

curl http://127.0.0.1:8055/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen3.6-35B-A3B-oQ8e-fp16-mtp", "messages": [{"role": "user", "content": "Hello"}]}'

Sampling follows Qwen guidance: thinking mode temperature 0.6, top_p 0.95; non-thinking temperature 0.7, top_p 0.8; top_k 20, min_p 0 throughout. Avoid greedy decoding with thinking enabled — Qwen documents that the combination can produce endless repetition.

Verified with oMLX. Loading outside oMLX (mlx-lm / mlx-vlm) requires a runtime that honours per-module quantization overrides in config.json; this has not been tested here.

License

Apache 2.0, inherited from Qwen/Qwen3.6-35B-A3B. A copy of the license is included as LICENSE. Quantization does not change the licensing terms — all credit for the underlying model belongs to the Qwen team.