Qwen3.6-35B-A3B-MTP-Preserved-GPTQ-Int4

Qwen3.6-35B-A3B — GPTQ Int4, MTP-preserved (clean)

GPTQ 4-bit quantization of the official, unmodified Qwen/Qwen3.6-35B-A3B that keeps all 19 mtp.* multi-token-prediction tensors in BF16, so vLLM's MTP speculative decoding works. Standard GPTQ exports strip these tensors; the only prior MTP-preserved quant was based on an abliterated variant. This one is the plain official model — no ablation, no fine-tuning.

Quantization

• GPTQModel 7.1.0 — bits=4, group_size=128, sym=true, desc_act=false, act_group_aware=true, damp=0.05, 256 C4 calibration samples @ BF16
• Excluded from quantization (kept BF16): mtp.* (draft head), MoE router gates
• Calibrated on an Intel Arc Pro B70 (XPU); ~21 GB output

Measured performance (single Arc Pro B70, 32 GB, single stream)

Served with vLLM XPU (intel/llm-scaler-vllm:0.21.0-b2 + two patches from the intel-arc-pro-b70-inference-cookbook), --speculative-config '{"method":"mtp","num_speculative_tokens":1}':

• ~150 t/s decode (short context) → ~92 t/s @ 8K context
• ~8.7K t/s prefill @ 4–8K prompts
• 94–97% MTP draft acceptance

Serving example

vllm serve <this-repo> --quantization gptq --dtype float16 \
  --max-model-len 16384 --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 --max-num-batched-tokens 8192 --language-model-only \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

On Intel Arc (XPU), GPTQ MoE currently needs the BF16-draft + native-int4 routing patches from the cookbook repo (two of its four patches are already upstreamed in llm-scaler-vllm 0.21.0-b2).

Credits

• Base model: Qwen team (Apache 2.0)
• MTP-preservation recipe: SergiioB's cookbook and llmfan46's original MTP-preserved quant (this repo redoes it from the clean official checkpoint)