Qwen3.6-35B-A3B-Hi-Fi-MTP-runtime

Qwen 3.6 35B-A3B Hi-Fi · MTP runtime variant

~76.0% draft acceptance (95% CI 75.7–76.3%) and ~1.49× decode speedup (95% CI 1.46–1.52×) in llama-server native speculative decoding, on a 50-prompt chat-template suite, A100-80GB, k=4, 3-seed lock (seeds 42, 1337, 2024).

Same calibration as the Hi-Fi Q4_K_M variant, with the MTP draft head (blk.40) retained. The MTP block inherits the Hi-Fi main-model quant policy plus one Q8_0 override on nextn.eh_proj; the draft head itself is not calibrated (see Scope).

🖥️ Datacenter target (A100-80GB). The speedup is a datacenter-GPU result. On Apple Silicon / consumer GPUs there is no measured speedup; for local runs use the Hi-Fi GGUF.

Requirements

• llama-server with Qwen 3.5/3.6 NextN MTP loader (llama.cpp main commit 2f6c815dc… or later)
• Send requests via the /v1/chat/completions endpoint (NOT /completion; see Scope)

How to run

llama-server \
  -m qwen36-35b-a3b-hi-fi-mtp-runtime.gguf \
  -ngl 999 -c 4096 \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 --spec-draft-n-min 1

Request body:

{
  "messages": [{"role": "user", "content": "Write a Python function ..."}],
  "max_tokens": 256,
  "temperature": 0,
  "speculative": {"type": "draft-mtp", "n_max": 4, "n_min": 1}
}

Inspect acceptance via timings.draft_n (proposed) and timings.draft_n_accepted (accepted).

Measured numbers (3-seed lock, 2026-05-29)

Vanilla (non-spec) decode baseline: 133.61 tok/s. Hardware: A100-80GB. Context: 4K. n_predict: 64. Greedy (temp=0, top_k=1).

Per-seed: seed 42 → 75.97%/1.472× · seed 1337 → 76.14%/1.502× · seed 2024 → 75.85%/1.493×.

Draft-window sensitivity

k=4 is the measured optimum. At higher k, per-position acceptance falls because deeper drafts compound on the head's own previous predictions; at k=3 the draft window is underused.

Scope

• Invocation. Raw prompts to /completion (no chat template) measure ~63% acceptance and 1.30× speedup. Use /v1/chat/completions or apply the Qwen chat template manually.
• Hardware. The 1.49× speedup is measured on A100-80GB at 4K context. On consumer hardware (Apple Silicon / M-series, single consumer GPUs) the A3B MoE is compute-bound and speculative verification yields no measured net speedup; for local / Apple Silicon use, run the Hi-Fi GGUF (same calibration, no MTP head, no special runtime). Longer contexts are unmeasured and may shift the speedup.
• Runtime. Requires llama-server. llama-cpp-python (as of 2026-05-29) does not expose the MTP draft path. Bare llama-cli does not run server-side speculative decoding.
• Calibration. The MTP block uses the main-model imatrix, which has no activation statistics for blk.40. Whether a calibrated draft head changes acceptance is unmeasured.

Provenance

• Base model: unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF BF16 revision 5c2410d71524f4f72b023ce8daf7a80528226d5f (MTP-inclusive)
• Imatrix: imatrix_codemath_v1_256k.dat (sha 5872a78f610050d2fccdce0c13ae450a472647c9fb297fe0a7ccaf2dfa945460), calibrated on the code+math corpus, 256K tokens
• llama.cpp: main commit 2f6c815dc450106ef877ae32f4472bfd5cf83e47
• Artifact sha256: 457114c45cd1b918ce71cfe01dcc7f4f70a0853092c911d9c6760ef16e25443c

All measurements and sensitivity probes are reproducible from the receipts under receipts/.

Related

• Hi-Fi (main model): fraQtl/Qwen3.6-35B-A3B-Hi-Fi-GGUF: same calibration, no MTP head.
• D1 (Mistral 7B fraQtl sidecars): fraQtl/Mistral-7B-v0.3-fraqtl-sidecars.

License

Inherits Qwen 3.6 license from the upstream base model.

More from fraQtl

Serving lane, KV-cache compression sidecars for vLLM: nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.