Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4

Qwen3.6-27B DSV4Pro Thinking Distill — NVFP4 (multi-tier + official MTP)

NVFP4 quantization of 27B DSV4Pro Thinking Distill with multiple tiers + official nextn MTP head, balancing quality, speed, and concurrency via W4A16 / W4A4 (/ future W4A8) + MTP.

Tiers

• W4A16 (MLP-focused): only MLP gate/up/down → NVFP4; attention, Mamba-GDN, vision, embeddings, lm_head, norms stay BF16.
• W4A4: additionally quantizes attention + GDN proj; only conv1d/lm_head/vision/embed/norm/MTP stay BF16.
• Both ship the official MTP head.

MTP (official nextn head) — all speeds @ RTX PRO 6000 Blackwell (sm120), vLLM 0.23

⚠️ Speeds are @R6000 + vLLM. The GGUF repo's MTP numbers are @Spark GB10 + llama.cpp — different hardware, don't compare directly.

MTP stays net-positive under concurrency (W4A4 N=2, aggregate tok/s): c1 +72% · c4 +58% · c8 +48% · c16 +44%. Gain decreases with concurrency but stays positive through c16 (NVFP4 is memory-bound-wide + sm120 compute-strong; crossover > 16). Use MTP for single-stream / low-to-mid concurrency.

Load

# W4A16 (root) + MTP
vllm serve nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --quantization modelopt \
  --max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":4}'

# W4A4 (w4a4/ subdir) + MTP
huggingface-cli download nerkyor/Qwen3.6-27B-DSV4Pro-Thinking-Distill-NVFP4 --local-dir ./nvfp4
vllm serve ./nvfp4/w4a4 --quantization modelopt \
  --max-model-len 32768 --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

Inference: temperature=0.6, top_p=0.95, max_tokens=32000 (thinking mode). Hardware: Blackwell SM120+ (RTX PRO 6000 / B200). SGLang: --quantization modelopt_fp4 + --speculative-algorithm NEXTN --speculative-num-steps N (optimal num-steps TBD on Spark).

W4A8 (pending vLLM support)

ModelOpt W4A8_NVFP4_FP8_CFG quantizes fine (~19 GB, quality expected ≈ W4A16), and R6000 sm120 has native FP8×FP4 mixed MMA. But vLLM 0.23's ModelOpt loader rejects W4A8_NVFP4_FP8 (accepts only FP8 / FP8_PER_CHANNEL_PER_TOKEN / FP8_PB_WO / NVFP4 / W4A16_NVFP4 / MXFP8 / MIXED_PRECISION). A w4a8/ subdir will be added once an engine supports it.

Eval protocol

temp 0.6 / top_p 0.95 / thinking-on / max_tokens 32000 / ctx 36864; GPQA-Diamond 198 (full) + MMLU-500 5-shot. Quality is hardware-independent; speed numbers are @R6000.

---

中文说明

NVFP4 多档量化 + 官方 MTP 头，通过 W4A16 / W4A4（/ 未来 W4A8）+ MTP 兼顾质量、速度、并发。

MTP 峰值 @R6000：W4A16 N=4→93.7 tok/s（2.08×）、W4A4 N=2→111 tok/s（1.76×）。MTP 并发全程正收益（W4A4 c16 仍 +44%，不转负）。所有速度 @ RTX PRO 6000（sm120）vLLM，区别于 GGUF 仓的 Spark GB10 数据。加载命令见上方英文部分。

Claude Code（实验性 / experimental）

Claude Code 支持目前属于实验性接入。Claude Code 需要兼容 Anthropic /v1/messages 的服务端和稳定的工具调用；直接使用 OpenAI-compatible chat endpoint 可能无法正常工作，需要桥接或兼容运行时。

本模型使用 Qwen3 XML 工具调用格式（<tool_call><function=name><parameter=...>）。vLLM 路线：

vllm serve /path/to/model \
  --served-model-name qwen36-27b-distill \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

Claude Code 指向 served-model-name（不要用带 / 的 HF repo id）；或使用 LM Studio 0.4.1+（内置 Claude Code /v1/messages）。参考 vLLM Claude Code · LM Studio。

---

Claude Code (experimental)

Claude Code support is experimental. It needs an Anthropic-compatible /v1/messages endpoint and stable tool calling; a direct OpenAI-compatible chat endpoint may not work and requires a bridge or compatible runtime. The model uses the Qwen3 XML tool format — on vLLM use --tool-call-parser qwen3_xml (command above). Point Claude Code at the served-model-name (no slashes), or use LM Studio 0.4.1+ (built-in Claude Code /v1/messages).