Qwen3.6-35B-A3B-NVFP4-DSPARK

Qwen3.6-35B-A3B-NVFP4 · DSPARK draft model

⚡ v2 is out — Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2: +6.8% acceptance across every Spec-Bench domain (macro 3.70 → 3.96), refreshed benchmarks on vLLM / sglang / llama.cpp, GGUF builds (-v2-GGUF) and prebuilt llama.cpp binaries (speed.llama.cpp). This v1 checkpoint remains available for anyone pinned to it; all new deployments should use v2.

Overview

A DSPARK speculative-decoding draft for unsloth/Qwen3.6-35B-A3B-NVFP4 (the Unsloth NVFP4 quantization of Qwen3.6-35B-A3B, a hybrid attention/gated-DeltaNet MoE). The draft consumes hidden states captured from that exact checkpoint, so it is matched to the NVFP4 weights as served — not to the bf16 original.

⚠️ Serve the frozen target Koopah/Qwen3.6-35B-A3B-NVFP4, not upstream main. On 2026-07-10 the upstream repo silently replaced its weights with a mixed-precision re-export that produces degenerate looping output in our testing and is NOT the checkpoint this draft was trained against. The frozen repo is byte-identical to upstream revision 612d523c58 — the artifact this card's numbers were validated with.

DSPARK drafts a whole block of tokens in a single forward pass and the target verifies them with the lossless ratio-acceptance rule (min(1, q/p)) — output quality is identical to running the target model alone, at any sampling temperature.

Supported targets: measured on both NVFP4 and NVFP4-Fast (acceptance actually improves on Fast — see the -Fast section). fp8 and bf16 variants of Qwen3.6-35B-A3B are untested but likely compatible: the draft reads the same hidden-state interface, and the Fast result shows cleaner activations help rather than hurt.

🚧 Training is still in progress. This is the best-scoring checkpoint from an ongoing multi-week run, published early. Acceptance is still climbing — expect updated weights.

Performance at a glance (single GPU)

Benchmark protocol matched to the sglang DSpark release post: one batch of B simultaneous requests, the post's fixed prompt (16 concatenated GSM8K questions), output forced to exactly 1024 tokens, temperature 0.7, prompt caching disabled, every point averaged over 3 rounds. Per-user decode speed is the median per-request rate from first token to completion; aggregate = per-user × B (steady state). One RTX PRO 6000 Blackwell 96 GB (Max-Q).

sglang (fork branch below)

* benchmarked without confidence-scheduled block truncation — no engine currently wires the draft's confidence head into inference (see known issues), so every round verifies the full drafted block. These numbers are a floor.

vLLM 0.26

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

Headlines: 3.1× solo / 1.4× at-capacity over plain decoding on vLLM (568 tok/s single-user, 6,835 tok/s aggregate), ahead of the built-in MTP head at every batch size on both engines. The k=4/6/8 family doubles as a static simulation of confidence scheduling: a wired confidence head would pick the verify budget per request per step, so the best-k envelope is a floor on what it would deliver (see known issues — no engine wires the head yet; all numbers verify blindly).

Continuous serving (closed loop)

The same GSM8K workload served as a continuous request stream (EOS-terminated generations, prompts sampled per request — closer to production traffic than the one-batch protocol above; sglang fork, radix cache on as in production):

Under sustained load with request churn, DSpark's margin over MTP is larger than in the one-batch numbers (+61% aggregate at capacity): continuous serving mixes prefill into every decode window, and DSpark's faster turnarounds keep queues shorter.

Evaluation detail

Serving-faithful rollout on held-out prompts never seen in training (summarization-heavy workload), sampling at temperature 1.0, top-p 0.95, top-k 20, ratio acceptance, no confidence threshold, bonus token included:

Per-position acceptance is nearly flat across all 8 positions — deep draft positions hold up as well as shallow ones, which is what makes the long block pay off.

Spec-Bench (6 domains, 480 prompts)

Measured end-to-end on the live sglang server (fork branch below), ratio acceptance, 8 draft positions, this checkpoint. Two conditions: temp 0 (greedy target — the convention most published tables use) and temp 1.0 + top-p 0.95 + top-k 20 (the sampled production serving condition):

Both columns will keep moving up while training continues.

The -Fast variant

The draft also works on unsloth/Qwen3.6-35B-A3B-NVFP4-Fast — without retraining, acceptance transfers intact (mean acceptance length 7.4 vs 6.2 on the training target, per-position 0.968/0.896/0.841/0.809/…, while the built-in MTP head is unchanged between the two targets — so the gain is the Fast recipe handing the draft cleaner capture-layer activations, not an easier target overall). Same benchmark protocol as above, vLLM 0.26:

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

Note: the -Fast variant currently serves via vLLM ≥ 0.26 only — the pinned sglang fork branch predates upstream support for the Fast recipe's quantization layout and fails to load it.

Deployment (sglang)

Use the pre-patched engine — upstream engines have several known issues with this exact target (detailed below):

• Patched fork branch: KoopahTManiac/sglang@dspark-hybrid-nvfp4 — last known-good upstream commit for NVFP4-on-Blackwell, plus four commits on top: the hybrid-GDN state-commit fix for DSPARK, a backport of upstream's DSPARK grammar-constrained decoding (#31753) — so JSON-schema / regex / EBNF output and tool_choice work through speculative decoding — and a backport of the NEXTN mamba-track guard (#27998), so the built-in MTP head also serves on this branch for A/B comparisons. Diff it yourself.
• A ready-to-build Dockerfile in this repo: a thin overlay on the official lmsysorg/sglang:nightly-dev-20260714-2f79d334 image (the last engine era before the SM120 NVFP4 regression, with era-matched kernels and the flashinfer JIT cache prebaked) that swaps in the fork branch's Python tree — all fork commits are pure Python. The default command carries the full tuned flag set and env; validated on an RTX PRO 6000 Blackwell: containerized serving matches native throughput within 1% (after first-minute JIT warmup).

# prebuilt image (or build it yourself from ./docker):
docker pull koopahtmaniac/sglang-dspark:latest   # pinned: :pin-0c01971ee
docker run -d --runtime nvidia --gpus all -p 30000:30000 koopahtmaniac/sglang-dspark
# or with local model/draft directories instead of HF downloads:
docker run -d --runtime nvidia --gpus all \
  -v /path/to/target-model:/models:ro -v /path/to/draft:/draft:ro \
  -e MODEL=/models -e DRAFT=/draft -p 30000:30000 koopahtmaniac/sglang-dspark

Full launch command as benchmarked (single ~96 GB Blackwell). The last four flags before the parsers are the high-concurrency tuning — --mem-fraction-static 0.74 pairs with the GDN state pool (--max-mamba-cache-size 540, admits ~96 concurrent) — drop those four for low-concurrency use:

python -m sglang.launch_server \
  --model-path Koopah/Qwen3.6-35B-A3B-NVFP4 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK \
  --tp 1 \
  --context-length 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --linear-attn-decode-backend flashinfer \
  --mamba-ssm-dtype bfloat16 \
  --chunked-prefill-size 16384 \
  --enable-mixed-chunk \
  --num-continuous-decode-steps 2 \
  --enforce-shared-experts-fusion \
  --enable-fused-qk-norm-rope \
  --mem-fraction-static 0.74 \
  --max-mamba-cache-size 540 \
  --cuda-graph-max-bs-decode 96 \
  --max-running-requests 96 \
  --grammar-backend xgrammar \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --trust-remote-code

TTFT at these settings (open loop, ~1k-token prompts): p50 ≈ 105 ms from 2 up to 10 req/s, p99 ≤ 215 ms at 10 req/s.

Deployment (vLLM)

vLLM 0.26 is currently the fastest way to serve this draft (see the chart and table in Performance at a glance). Requirements and tuning:

• vLLM ≥ 0.26.0 — older versions fail to load this draft (KV page-size unification and draft-config handling both fixed in 0.26).
• --mamba-ssm-cache-dtype bfloat16 matters: without it the GDN recurrent states default to fp32 and DSpark's deep verify staging caps admission at ~47 concurrent requests (high-batch aggregate drops ~25%).
• Use the default KV-cache dtype (fp8 KV crashes on this hybrid target, see known issues).

vllm serve Koopah/Qwen3.6-35B-A3B-NVFP4 \
  --trust-remote-code \
  --served-model-name qwen3.6-35b-a3b \
  --max-num-seqs 96 --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.9 \
  --mamba-ssm-cache-dtype bfloat16 \
  --speculative-config '{"method": "dspark", "model": "Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK", "num_speculative_tokens": 8}'

⚠️ Known issues in upstream engines (as of July 2026) — why the fork exists

1. Hybrid-GDN state commit (affects this target!): upstream sglang's DSPARK worker does not yet commit the target's gated-DeltaNet state after speculative verification. On hybrid-linear-attention targets like Qwen3.6-35B-A3B this silently collapses acceptance and corrupts outputs. The fix is already included in the fork branch and Docker image above (a port of DFlash's _update_target_mamba_state_after_verify into the DSPARK worker); an upstream PR will follow. Dense targets are unaffected.
2. NVFP4 on SM120 (consumer Blackwell): sglang's FP4 quantization refactor (9756f768a) routes nvfp4-pack-quantized compressed-tensors MoE onto a FlashInfer kernel path with no working SM12x support — garbage output at any temperature. Upstream work is in flight (sglang #32119, flashinfer #3932); until it lands, use the pinned fork branch above (0c01971ee base, the last known-good).
3. vLLM: fp8 KV cache crashes on this hybrid target — use the default KV dtype.
4. Confidence head unused by both engines: this draft ships a trained confidence head for dynamic block truncation (measured on-policy calibration: ECE 0.010, AUC 0.925), but vLLM explicitly skips its weights ("not wired into inference yet") and sglang's confidence-scheduling only supports MLA-attention targets. All benchmark numbers on this card therefore verify the full block every round — once either engine wires the head, expect the mid-batch throughput gap vs MTP to close and the peaks to rise.

Model Specifications

Training Details

• Data: ~1.34 M prompts (open-perfectblend mix) with responses regenerated by the target itself in both thinking and non-thinking modes, sampled at the target's serving configuration — the draft learns the target's actual output distribution rather than generic text.
• Stack: DeepSpec, adapted for online single-GPU training against the live NVFP4 target.
• Initialization: warm-started from the z-lab DFlash backbone for this target, converted to DSPARK format (Markov head soft-started, confidence head fresh), then trained from step one under the DSPARK objective.
• Evaluation during training: rollout-based accept-length eval on held-out prompts every 100 steps at the serving sampling configuration; this repo tracks the best-scoring checkpoint.

References

• DSpark: block-diffusion speculative decoding (DeepSeek, 2026) — merged in sglang and vLLM
• DeepSpec training framework
• z-lab DFlash (backbone initialization)

Run your own benchmark

This repo includes eval_throughput.py — a portable, single-file benchmark for any OpenAI-compatible endpoint. It measures decode throughput by content category, decode speed vs context depth, and batch scaling, and renders charts (matplotlib optional).

pip install requests matplotlib
# with the DSpark drafter enabled:
python eval_throughput.py --base-url http://127.0.0.1:30000/v1 --label dspark
# restart your server without speculative decoding, then:
python eval_throughput.py --base-url http://127.0.0.1:30000/v1 --label baseline
# compare eval_results/results_*.json / charts_*.png for your speedup

Pass --api-key (or set OPENAI_API_KEY) if your server requires one, and --quick for a fast smoke run. Numbers vary with GPU, sampling temperature, and context depth — expect acceptance (and speedup) to decline on long contexts; see the notes above.