Sparkulator-Qwen3.6-27B

Sparkulator-Qwen3.6-27B

An NVFP4 re-quantization of the MTP speculative-decoding head inside nvidia/Qwen3.6-27B-NVFP4, built to make speculative decoding cheaper on an NVIDIA DGX Spark (GB10, 128 GB unified LPDDR at ~273 GB/s). It is a community artifact from How To Spark — not an official NVIDIA or Qwen release.

Speculative decoding is verified, so the draft cannot change what the target model outputs — only how fast it gets there. That is why quantizing it aggressively is safe: the only thing at risk is the draft's acceptance rate, and here acceptance held.

What this is

nvidia/Qwen3.6-27B-NVFP4 is a mixed-precision ModelOpt export: the MLPs and lm_head are NVFP4, all attention is FP8 — and the 15 mtp.* tensors of the native MTP draft head are left BF16, listed in the checkpoint's own ignore list. On a dense 27B target that draft is 0.791 GiB, and at the recipe's tuned num_speculative_tokens: 5 it is read five times per accepted step. That makes it roughly a fifth of the bytes moved per decode step — the largest un-quantized thing left in an otherwise-quantized checkpoint.

This repo holds those 8 linears re-quantized to W4A16 NVFP4, group size 16 — the exact scheme and on-disk layout ModelOpt already uses for the MLPs in the same file, so vLLM loads it through the existing path with no patch.

Quantized: mtp.fc, mtp.layers.0.self_attn.{q,k,v,o}_proj, mtp.layers.0.mlp.{gate,up,down}_proj. Left BF16: all norms (input_layernorm, post_attention_layernorm, q_norm, k_norm, norm, pre_fc_norm_hidden, pre_fc_norm_embedding). The draft ships no lm_head or embed_tokens of its own — vLLM shares the target's, which are already NVFP4.

Measured on two DGX Sparks (2026-07-27)

vLLM 0.26.0, single node, k=5, --max-num-seqs 4, full 262,144-token context, greedy, concurrency 1, code workload, thinking off, warm. Counterbalanced: Spark-1 ran NVFP4-draft → BF16, Spark-2 ran BF16 → NVFP4. Medians over 6/10/6 repeats.

Six of six arms favour the quantized draft. The size of the win matches the byte census: dropping 0.57 GiB from a draft that is read 5x per step removes ~2.8 GiB from a ~25 GiB step, i.e. ~11% — and the measured median gain is ~+10%.

Acceptance held. On a fixed 3-prompt set (identical requests in both arms): 70.60% → 71.29% on Spark-1, 69.85% → 67.85% on Spark-2. Over the benchmark windows: 62.00% → 61.57% and 61.04% → 61.92%. Data-free RTN cost this draft nothing measurable — the speed came from bytes, not from drafting better.

One caveat, stated plainly. A third boot of this configuration on Spark-2 produced a sustained stall — individual requests at 8–17 tok/s against a ~30 tok/s median, per-request sd 6.0 where every other boot measured 0.5–2.4 — and its medians (34.39 / 27.95 / 15.52) would have read as a large regression. It did not reproduce on a re-boot of the identical configuration and the pathology is visible in the per-request samples, so the table above uses the clean boot. If you benchmark this yourself and see numbers below the BF16 arm, re-boot the server before believing them.

Using it

Graft the tensors back onto the base checkpoint (they are a drop-in replacement for the BF16 mtp.* tensors in model-00003-of-00003.safetensors), and merge quantization_config.json into the checkpoint's config.json:

# see quantize_mtp_nvfp4.py — it rebuilds the whole variant directory from the
# base checkpoint in about a minute, hardlinking the two shards it does not touch.
python3 quantize_mtp_nvfp4.py \
  --src ~/models/hf/Qwen3.6-27B-NVFP4 \
  --out ~/models/hf/Qwen3.6-27B-NVFP4-MTP-NVFP4

Then serve the variant directory exactly as you would the original:

vllm serve ~/models/hf/Qwen3.6-27B-NVFP4-MTP-NVFP4 \
  --served-model-name qwen3.6-27b \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-memory-bytes 15032385536 \
  --max-num-seqs 4 --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Confirm it actually loaded: Model loading took 20.24 GiB (not 20.8). If it reads 20.8 the packed tensors were skipped and you are running the BF16 draft.

The one thing that will bite you

This checkpoint carries its quantization config in two places — config.json's quantization_config and the legacy hf_quant_config.json — and vLLM prefers config.json. Patch only hf_quant_config.json and vLLM builds mtp.fc unquantized, then dies at load with assert self.data.shape == loaded_weight.shape. Both files must declare the 8 layers W4A16_NVFP4 and drop mtp* from the exclude/ignore list.

Credit

The base model is Qwen/Qwen3.6-27B (Apache-2.0); the mixed-precision NVFP4 export this is built from is nvidia/Qwen3.6-27B-NVFP4 by NVIDIA ModelOpt. All this repo adds is a quantization of the draft head they left in BF16, plus the measurement of what that is worth on a Spark. The deployment recipe lives at howtospark.com/recipes/qwen3-6-27b-nvfp4.