Qwen3.6-35B-A3B-NVFP4-DSPARK-v2

Qwen3.6-35B-A3B-NVFP4 · DSPARK draft model (v2)

v2 — a later training checkpoint than v1, +6.8% acceptance across the board (Spec-Bench macro 3.70 → 3.96, every domain improved). v1 remains available at the link above for anyone pinned to it. Training continues; a final version will follow.

Overview

A DSPARK speculative-decoding draft for unsloth/Qwen3.6-35B-A3B-NVFP4 (the Unsloth NVFP4 quantization of Qwen3.6-35B-A3B, a hybrid attention/gated-DeltaNet MoE). The draft consumes hidden states captured from that exact checkpoint, so it is matched to the NVFP4 weights as served — not to the bf16 original.

⚠️ Serve the frozen target Koopah/Qwen3.6-35B-A3B-NVFP4, not upstream main. On 2026-07-10 the upstream repo silently replaced its weights with a mixed-precision re-export that produces degenerate looping output in our testing (two SM120 GPUs, all KV dtypes) and is NOT the checkpoint this draft was trained against. The frozen repo is byte-identical to upstream revision 612d523c58 — the artifact every number on this card was validated with.

DSPARK drafts a whole block of tokens in a single forward pass and the target verifies them with the lossless ratio-acceptance rule (min(1, q/p)) — output quality is identical to running the target model alone, at any sampling temperature.

Supported targets: measured on both NVFP4 and NVFP4-Fast (acceptance transfers intact — see the -Fast section). fp8 and bf16 variants of Qwen3.6-35B-A3B are untested but likely compatible: the draft reads the same hidden-state interface, and the Fast result shows cleaner activations help rather than hurt.

🚧 Training is still in progress. This is the best-scoring checkpoint from an ongoing multi-week run, published early. Acceptance is still climbing — expect updated weights.

Performance at a glance (single GPU)

Benchmark protocol matched to the sglang DSpark release post: one batch of B simultaneous requests, the post's fixed prompt (16 concatenated GSM8K questions), output forced to exactly 1024 tokens, temperature 0.7, prompt caching disabled, every point averaged over 3 rounds. Per-user decode speed is the median per-request rate from first token to completion; aggregate = per-user × B (steady state). One RTX PRO 6000 Blackwell 96 GB (Max-Q).

sglang (fork branch below)

* benchmarked without confidence-scheduled block truncation — no engine currently wires the draft's confidence head into inference (see known issues), so every round verifies the full drafted block. These numbers are a floor.

vLLM 0.26

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

Headlines: 3.1× solo / 1.5× at-capacity over plain decoding on vLLM (580 tok/s single-user, 7,229 tok/s aggregate), ahead of the built-in MTP head at every batch size on both engines. The k=4/6/8 family doubles as a static simulation of confidence scheduling: a wired confidence head would pick the verify budget per request per step, so the best-k envelope is a floor on what it would deliver (see known issues — no engine wires the head yet; all numbers verify blindly).

llama.cpp (single-user / local serving)

DSpark also serves through llama.cpp — GGUF conversions of this draft are in Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-GGUF. Upstream llama.cpp implements the DSPARK spec type; the speed.llama.cpp performance branch (upstream + fused in-graph truncation sampler, MoE finalize / weight-load fusions, batched elementwise launches, host-path fixes, and exact ratio acceptance for sampled drafting) is +82% over the upstream base on this spec path and is what the numbers below use. Output distribution verified identical to non-speculative serving (N=3200/config total-variation test at temp 0.7).

Same protocol as above, llama-server, BF16 drafter, dedicated single-user config (-np 1):

(Upstream measured under the identical protocol and config — notably, upstream gains nothing from -np 1, since its per-round overhead dominates regardless of slot count; the speed series is what makes single-slot serving pay.)

llama.cpp wins plain single-stream decoding outright and reaches ~87% of vLLM with speculation — the remaining gap is per-round host/launch overhead, not acceptance (accept lengths match or beat vLLM's on this workload). Batch serving is currently host-bound in llama-server: the raw engine scales to 1,900 tok/s at bs64 (llama-batched-bench), but the server's single-threaded per-slot loop (host sampling, stream writes) caps served throughput near 700 aggregate regardless of k — use vLLM/sglang at capacity, llama.cpp for single-user/local. Two deploy notes: pass --cache-ram 0 at high slot counts (the host prompt-cache eviction path can livelock with this model's large per-slot state), and size -ub ≥ slots × 8 when running many parallel slots (the draft block is non-causal).

Verification (SPEED-Bench, llama.cpp)

Per-category verification on nvidia/SPEED-Bench (qualitative split, all samples, temp 0, OSL 1024, single user) via the in-tree speed-bench harness: baseline llama-server vs DSpark k=8 (BF16 drafter), speed.llama.cpp branch:

Continuous serving (closed loop)

The same GSM8K workload served as a continuous request stream (EOS-terminated generations, prompts sampled per request — closer to production traffic than the one-batch protocol above; sglang fork, radix cache on as in production):

Under sustained load with request churn, DSpark's margin over MTP is larger than in the one-batch numbers (+61% aggregate at capacity): continuous serving mixes prefill into every decode window, and DSpark's faster turnarounds keep queues shorter.

Evaluation detail

Serving-faithful rollout on held-out prompts never seen in training (summarization-heavy workload), sampling at temperature 1.0, top-p 0.95, top-k 20, ratio acceptance, no confidence threshold, bonus token included:

Per-position acceptance is nearly flat across all 8 positions — deep draft positions hold up as well as shallow ones, which is what makes the long block pay off.

Spec-Bench (6 domains, 480 prompts)

Serving-faithful accept length (sglang fork, ratio acceptance, 8 draft positions, temp 1.0 top-p 0.95 top-k 20). v2 vs v1, same harness:

Every domain improved (+6.8% macro) — the reason v2 exists. Numbers keep climbing while training continues.

The -Fast variant

The draft also works on unsloth/Qwen3.6-35B-A3B-NVFP4-Fast — without retraining, acceptance transfers intact: with v2 the mean acceptance length on -Fast matches the training target (≈6.0–6.1 vs ≈6.0–6.1 on this protocol's workload; with v1 the -Fast target actually accepted better, 7.4 vs 6.2 — v2's training has since closed that gap from the training-target side). Same benchmark protocol as above, vLLM 0.26:

* same disclaimer: measured without confidence-scheduled truncation — a floor, not a ceiling.

Note: the -Fast variant currently serves via vLLM ≥ 0.26 only — the pinned sglang fork branch predates upstream support for the Fast recipe's quantization layout and fails to load it.

Deployment (sglang)

Use the pre-patched engine — upstream engines have several known issues with this exact target (detailed below):

• Patched fork branch: KoopahTManiac/sglang@dspark-hybrid-nvfp4 — last known-good upstream commit for NVFP4-on-Blackwell, plus four commits on top: the hybrid-GDN state-commit fix for DSPARK, a backport of upstream's DSPARK grammar-constrained decoding (#31753) — so JSON-schema / regex / EBNF output and tool_choice work through speculative decoding — and a backport of the NEXTN mamba-track guard (#27998), so the built-in MTP head also serves on this branch for A/B comparisons. Diff it yourself.
• A ready-to-build Dockerfile in this repo: a thin overlay on the official lmsysorg/sglang:nightly-dev-20260714-2f79d334 image (the last engine era before the SM120 NVFP4 regression, with era-matched kernels and the flashinfer JIT cache prebaked) that swaps in the fork branch's Python tree — all fork commits are pure Python. The default command carries the full tuned flag set and env; validated on an RTX PRO 6000 Blackwell: containerized serving matches native throughput within 1% (after first-minute JIT warmup).

# prebuilt image (or build it yourself from ./docker):
docker pull koopahtmaniac/sglang-dspark:latest   # pinned: :pin-0c01971ee
docker run -d --runtime nvidia --gpus all -p 30000:30000 koopahtmaniac/sglang-dspark
# or with local model/draft directories instead of HF downloads:
docker run -d --runtime nvidia --gpus all \
  -v /path/to/target-model:/models:ro -v /path/to/draft:/draft:ro \
  -e MODEL=/models -e DRAFT=/draft -p 30000:30000 koopahtmaniac/sglang-dspark

Full launch command as benchmarked (single ~96 GB Blackwell). The last four flags before the parsers are the high-concurrency tuning — --mem-fraction-static 0.74 pairs with the GDN state pool (--max-mamba-cache-size 540, admits ~96 concurrent) — drop those four for low-concurrency use:

python -m sglang.launch_server \
  --model-path Koopah/Qwen3.6-35B-A3B-NVFP4 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2 \
  --tp 1 \
  --context-length 32768 \
  --kv-cache-dtype fp8_e4m3 \
  --linear-attn-decode-backend flashinfer \
  --mamba-ssm-dtype bfloat16 \
  --chunked-prefill-size 16384 \
  --enable-mixed-chunk \
  --num-continuous-decode-steps 2 \
  --enforce-shared-experts-fusion \
  --enable-fused-qk-norm-rope \
  --mem-fraction-static 0.74 \
  --max-mamba-cache-size 540 \
  --cuda-graph-max-bs-decode 96 \
  --max-running-requests 96 \
  --grammar-backend xgrammar \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --trust-remote-code

TTFT at these settings (open loop, ~1k-token prompts): p50 ≈ 105 ms from 2 up to 10 req/s, p99 ≤ 215 ms at 10 req/s.

Deployment (vLLM)

vLLM 0.26 is currently the fastest way to serve this draft (see the chart and table in Performance at a glance). Requirements and tuning:

• vLLM ≥ 0.26.0 — older versions fail to load this draft (KV page-size unification and draft-config handling both fixed in 0.26).
• --mamba-ssm-cache-dtype bfloat16 matters: without it the GDN recurrent states default to fp32 and DSpark's deep verify staging caps admission at ~47 concurrent requests (high-batch aggregate drops ~25%).
• Use the default KV-cache dtype (fp8 KV crashes on this hybrid target, see known issues).

vllm serve Koopah/Qwen3.6-35B-A3B-NVFP4 \
  --trust-remote-code \
  --served-model-name qwen3.6-35b-a3b \
  --max-num-seqs 96 --max-num-batched-tokens 16384 \
  --gpu-memory-utilization 0.9 \
  --mamba-ssm-cache-dtype bfloat16 \
  --speculative-config '{"method": "dspark", "model": "Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2", "num_speculative_tokens": 8}'

Deployment (llama.cpp)

No conversion needed — prebuilt GGUFs (native-NVFP4 target + this draft) live in Koopah/Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-GGUF, and prebuilt binaries for Windows (CUDA 12.4/13.3, Vulkan, SYCL, HIP, CPU), Linux and macOS ship from the speed.llama.cpp releases. The performance branch is upstream llama.cpp (which implements DSPARK, PR #25173) plus the speed series — fused in-graph truncation sampler, MoE / weight-load fusions, batched launches, host-path fixes — and exact ratio acceptance for sampled drafting (the served distribution is verified identical to non-speculative sampling at any temperature; N=3200/config total-variation test). Performance and the upstream A/B are in the llama.cpp section above (506 tok/s single-user, +82% over the upstream base). A field report from a stock RTX 5090 desktop on the release binaries: 437–616 tok/s depending on content, at 128k context with q8 KV.

llama-server \
  -m  Qwen3.6-35B-A3B-NVFP4.gguf \
  -md Qwen3.6-35B-A3B-NVFP4-DSPARK-v2-BF16.gguf \
  --spec-type draft-dspark --spec-draft-n-max 8 \
  -ngl 99 -fa on -c 8192 -np 1

⚠️ Known issues in upstream engines (as of August 2026) — why the forks exist

1. Hybrid-GDN state commit (affects this target!): upstream sglang's DSPARK worker does not yet commit the target's gated-DeltaNet state after speculative verification. On hybrid-linear-attention targets like Qwen3.6-35B-A3B this silently collapses acceptance and corrupts outputs. The fix is already included in the fork branch and Docker image above (a port of DFlash's _update_target_mamba_state_after_verify into the DSPARK worker); an upstream PR will follow. Dense targets are unaffected.
2. NVFP4 on SM120 (consumer Blackwell): sglang's FP4 quantization refactor (9756f768a) routes nvfp4-pack-quantized compressed-tensors MoE onto a FlashInfer kernel path with no working SM12x support — garbage output at any temperature. Upstream work is in flight (sglang #32119, flashinfer #3932); until it lands, use the pinned fork branch above (0c01971ee base, the last known-good).
3. vLLM: fp8 KV cache crashes on this hybrid target — use the default KV dtype.
4. Confidence head unused in the vLLM/sglang numbers here: this draft ships a trained confidence head for dynamic block truncation (measured on-policy calibration: ECE 0.010, AUC 0.925). vLLM explicitly skips its weights ("not wired into inference yet"). At the time these benchmarks ran, sglang's confidence-scheduling supported MLA-attention targets only, so all vLLM/sglang numbers on this card verify the full block every round; sglang upstream has since (Aug 2026) generalized its confidence-fed ragged-verify scheduler beyond MLA — including hybrid GQA+mamba targets like this one — driven by offline-profiled SPS cost tables and STS calibration (--speculative-dspark-sps-table-path / --speculative-dspark-confidence-sts-path). Untested with this draft so far, and the path has open upstream issues (e.g. runtime- gamma and compact-graph geometry bugs), so treat it as promising but unverified. llama.cpp wires it: the llama.cpp deployment implements the DSpark paper's confidence scheduler using this head (--spec-sched, self-calibrating) — see the GGUF repo for deployment and usage guidance. vLLM/sglang integration of the head remains the open upside for mid-batch throughput vs MTP.

Model Specifications

Training Details

• Data: ~1.34 M prompts (open-perfectblend mix) with responses regenerated by the target itself in both thinking and non-thinking modes, sampled at the target's serving configuration — the draft learns the target's actual output distribution rather than generic text.
• Stack: DeepSpec, adapted for online single-GPU training against the live NVFP4 target.
• Initialization: warm-started from the z-lab DFlash backbone for this target, converted to DSPARK format (Markov head soft-started, confidence head fresh), then trained from step one under the DSPARK objective.
• Evaluation during training: rollout-based accept-length eval on held-out prompts every 100 steps at the serving sampling configuration; this repo tracks the best-scoring checkpoint.

References

• DSpark: block-diffusion speculative decoding (DeepSeek, 2026) — merged in sglang and vLLM
• DeepSpec training framework
• z-lab DFlash (backbone initialization)

Run your own benchmark

This repo includes eval_throughput.py — a portable, single-file benchmark for any OpenAI-compatible endpoint. It measures decode throughput by content category, decode speed vs context depth, and batch scaling, and renders charts (matplotlib optional).

pip install requests matplotlib
# with the DSpark drafter enabled:
python eval_throughput.py --base-url http://127.0.0.1:30000/v1 --label dspark
# restart your server without speculative decoding, then:
python eval_throughput.py --base-url http://127.0.0.1:30000/v1 --label baseline
# compare eval_results/results_*.json / charts_*.png for your speedup

Pass --api-key (or set OPENAI_API_KEY) if your server requires one, and --quick for a fast smoke run. Numbers vary with GPU, sampling temperature, and context depth — expect acceptance (and speedup) to decline on long contexts; see the notes above.