Qwen3.6-35B-A3B-oQ8-mtp

Qwen3.6-35B-A3B-oQ8-mtp

An oQ8 mixed-precision quantization of Qwen/Qwen3.6-35B-A3B for MLX, quantized with mlx-optiq, with Multi-Token Prediction (MTP) heads grafted on from the reference mtp.safetensors structure.

This repo also includes benchmark data comparing MTP on vs. off (MTP Performance) and quality benchmarks for the quantized checkpoint (Intelligence Benchmarks) below. Summary on MTP: in this setup, enabling it does not provide a token generation speedup, and in single-request decoding it is measurably slower.

Model Details

• Base model: Qwen/Qwen3.6-35B-A3B (Apache 2.0), a sparse MoE model with 35B total parameters and about 3B active parameters, with a 262,144 token native context extensible to about 1,010,000.
• Quantization method: mlx-optiq, oQ8, a mixed-precision quantization targeting about 8-bit average bits per weight, with sensitive layers kept at higher precision.
• MTP heads: Grafted from the official Qwen3.6 reference mtp.safetensors structure (37 tensors, group_size=64), reverse-engineered to match the base model's MTP layer layout so the quantized checkpoint can run with speculative or MTP decoding enabled.
• Format: MLX-native safetensors, quantized for use with an MLX inference server (tested with oMLX).
• Intended hardware: Apple Silicon with unified memory (developed and tested on an M4 Max, 128GB).

Usage

# with an oMLX-compatible server
omlx serve --model programmer-666/Qwen3.6-35B-A3B-oQ8-mtp

# or load directly with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("programmer-666/Qwen3.6-35B-A3B-oQ8-mtp")
response = generate(
    model, tokenizer,
    prompt="Explain multi-token prediction in one paragraph.",
    max_tokens=256,
)
print(response)

To run without MTP, disable speculative decoding in your server config. The base weights are unaffected, and inference falls back to standard autoregressive decoding.

MTP Performance

Benchmarks below compare this checkpoint with MTP enabled vs. MTP disabled, on the same hardware (M4 Max, 128GB unified memory), same prompt and generation lengths, and same inference server.

Single Request Decoding

• Token generation throughput (tg TPS) is consistently lower with MTP enabled, by an average of about 13.6% across context lengths, with the gap widening to 14 to 18% slower once context exceeds about 4K tokens.
• Peak memory is about 2.1% higher with MTP enabled across the board (the extra MTP head weights and their activation buffers).
• TTFT (prefill) is roughly a wash, about 0.3% average difference with no consistent direction. MTP heads don't participate in prefill, so this is expected and mostly run to run noise.

Continuous Batching

Under continuous batching, the gap mostly closes. MTP is within noise of the no-MTP baseline (-6.6% to +0.8%), with no consistent advantage at any batch size.

Takeaway

MTP head grafting is functional (weights load, model runs, output is correct), but in this MLX/oMLX inference setup, enabling MTP does not deliver a speculative decoding speedup, and for single-request decoding it does the opposite. This is most likely because the MTP head isn't yet wired into an actual speculative decode/verify loop in the serving stack; it currently just adds a forward pass and memory overhead without skipping any accept-reject cycle. Treat the MTP tensors in this repo as a structurally correct starting point for that integration work, not as a ready speedup.

If you get MTP-accelerated decoding working with this checkpoint on a different server or runtime, opening a discussion on this repo with your results would be very welcome.

Intelligence Benchmarks

Quality checks run on this quantized checkpoint with MTP disabled (thinking mode enabled for all runs). Equivalent runs with MTP enabled are in progress and will be added once complete; MTP only affects decoding strategy, not the model's weights, so these numbers reflect the checkpoint's underlying quality regardless of MTP status, but a direct on/off comparison isn't available yet.

These results are also embedded as structured model-index metadata in this README's YAML header, so they will surface in the Hub's evaluation results widget on the model page. Several benchmarks use a sampled subset rather than the full set, so treat MathQA and LiveCodeBench results (30 and 100 samples respectively) as indicative rather than tight estimates. MMLU-Pro, MBPP, and especially the full-set HumanEval run are more reliable.

Quantization to oQ8 preserves strong coding performance (HumanEval, MBPP) and general knowledge (MMLU-Pro) close to what's expected of the base model. LiveCodeBench, the hardest and most contamination-resistant of these, comes in noticeably lower, which lines up with it being the more demanding, competition-style coding benchmark rather than being oQ8-specific degradation. A same-precision base-model run on the same subset would be needed to isolate a quantization effect there.

Quantization Notes

• Quantized with mlx-optiq targeting oQ8 mixed precision. Comparable full-precision/oQ7 runs on the same base architecture landed around 37GB, so expect a similar footprint here (see Files and versions for exact sizes).
• No calibration or fine-tuning was performed beyond quantization and MTP head grafting; this is a direct weight-space conversion of the base model.

License

This is a derivative of Qwen/Qwen3.6-35B-A3B, released under the Apache 2.0 license. See the base model card for full terms, intended use, and limitations.