Qwen3.6-35B-A3B-MTP-Mixed-Q8_0

Qwen3.6-35B-A3B-MTP-Mixed-Q6_K

Overview

This repository provides a highly optimized, custom-quantized GGUF model of Qwen3.6-35B-A3B MTP, specifically engineered for extreme long-context local inference on Dual-GPU setups (2× RTX 3090, 24GB). The primary objective of this quantization is to maximize Multi-Token Prediction (MTP / Self-Speculative Decoding) efficiency while preserving the reasoning, multimodal, and long-context capacities of the original model.

Unlike my previous Qwen3.6-27B-MTP-Custom-IQ6_K project, this new release is entirely built from the official Unsloth MTP-ready BF16 GGUFs, which already integrate the corrected chat template and proper MTP metadata support. This allowed a much cleaner quantization pipeline while significantly improving speculative decoding behavior and overall throughput.

The architecture uses a carefully mixed quantization strategy:

• Core expert up/down tensors quantized to Q6_K
• Critical routing, attention, speculative decoding, and output tensors preserved in Q8_0
• Token embeddings and output layers preserved in Q8_0
• NextN speculative layers fully preserved in Q8_0

This quantization was specifically tuned to maintain extremely high draft acceptance rates while dramatically reducing VRAM usage compared to full Q8_0 deployments.

Base model used for requantization:

• unsloth/Qwen3.6-35B-A3B-MTP-GGUF

---

Engine Compatibility

Important

This model has been specifically designed and validated for the mtp-clean branch of llama.cpp maintained by am17an.

Mainline llama.cpp currently does not achieve comparable MTP performance and still lacks proper multimodal (mmproj) compatibility together with MTP speculative decoding. The mtp-clean branch currently provides:

• Proper MTP support
• Correct multimodal integration
• Much higher speculative decoding throughput
• Better tensor splitting behavior on dual GPU systems

Repository used:

git clone -b mtp-clean https://github.com/am17an/llama.cpp.git

As of today, using this branch is strongly recommended to reproduce the performance numbers presented in this repository.

---

Research & Methodology

Preserving Speculative Decoding (MTP)

Most standard quantization pipelines aggressively compress the speculative decoding layers, severely degrading draft acceptance rates and reducing MTP efficiency.

To avoid this, the following layers were explicitly preserved in Q8_0:

• nextn
• attention projections
• routing tensors
• gating tensors
• output tensors
• embeddings

Meanwhile, the large expert tensors were quantized to Q6_K to reduce memory footprint while preserving quality.

The exact quantization scheme was:

This configuration achieved exceptional speculative decoding efficiency even under extremely large context loads.

---

Performance

Using the recommended server configuration below, the following performance was achieved on a 262K context session:

prompt eval time = 185569.97 ms / 259716 tokens
                 = 0.71 ms per token
                 = 1399.56 tokens per second

eval time       = 29168.45 ms / 2428 tokens
                 = 12.01 ms per token
                 = 83.24 tokens per second

total time      = 214738.42 ms / 262144 tokens

draft acceptance rate = 0.93550
                      = 1581 accepted / 1690 generated

These results represent one of the highest publicly documented MTP acceptance rates currently achieved on a heavily quantized long-context deployment.

---

Chat Template & Metadata

Unlike my previous Qwen3.6-27B-MTP-Custom-IQ6_K project, this repository no longer required manual GGUF metadata patching.

The model was directly quantized from Unsloth's official MTP-ready BF16 release, meaning:

• Correct chat template already integrated
• Proper Developer role support
• Correct multimodal metadata
• Native MTP support
• No manual GGUF editing required

---

iMatrix Calibration

The model was calibrated using a custom shuffled iMatrix designed to preserve performance across:

• Coding tasks
• Tool usage
• Long-context reasoning
• English/French bilingual tasks
• Instruction following

The calibration dataset was built from merged subsets of:

• code_small
• tools_small
• text_en_small
• text_fr_small

from:

• eaddario/imatrix-calibration

---

Recommended Usage

To reproduce the optimal performance characteristics shown above, the following server configuration is recommended:

/path/to/llama.cpp/build/bin/llama-server \
    -m /path/to/Qwen3.6-35B-A3B-MTP-Mixed-Q6_K.gguf \
    --mmproj /path/to/mmproj-F16.gguf \
    --split-mode tensor \
    --tensor-split 1,1 \
    --host 0.0.0.0 \
    --port 8080 \
    --ctx-size 262144 \
    --parallel 1 \
    --gpu-layers 999 \
    --cache-type-k bf16 \
    --cache-type-v bf16 \
    --context-shift \
    --flash-attn on \
    -b 1024 -ub 1024 \
    --spec-type draft-mtp \
    --spec-draft-n-max 2 \
    --spec-draft-p-min 0.75 \
    --ctx-checkpoints 8 \
    --checkpoint-every-n-tokens 65536 \
    --cache-ram 32768 \
    --image-min-tokens 1024 \
    --alias Qwen3.6-35B \
    --jinja

---

Additional Notes

I also developed a custom proxy layer allowing users to dynamically switch between thinking and non-thinking behaviors while automatically applying the recommended sampling parameters together with "Preserve Thinking" support.

This proxy is available on my GitHub.

---

Hardware Requirements

• Target VRAM: 48 GB (validated on 2× RTX 3090 24GB)
• System RAM: Minimum 32GB recommended
• Recommended KV cache: F16
• Recommended context: 262K

The provided configuration uses a very large KV cache footprint. If you encounter OOM errors, consider:

• reducing --ctx-size

• lowering batch sizes

• using quantized KV cache:

  • --cache-type-k q8_0
  • --cache-type-v q8_0

---

Acknowledgments

This project was made possible thanks to the outstanding work of the open-source AI community. Special thanks to:

• Unsloth For publishing the official MTP-ready BF16 GGUF release with corrected metadata, proper chat templates, multimodal compatibility, and native speculative decoding support.

• am17an For the mtp-clean llama.cpp branch enabling high-performance multimodal MTP inference and substantially improved speculative decoding throughput compared to current mainline llama.cpp.

• eaddario For the excellent imatrix-calibration dataset used to build the multilingual and code-oriented custom iMatrix.

• The Qwen Team For researching and releasing the exceptional Qwen3.6 architecture and open-weight models enabling this work.