Qwen3.8-Flash-Next-NVFP4-DFlash

Qwen3.8-Flash-Next-NVFP4 — DFlash drafter

A DFlash speculative-decoding draft head for nvidia/Qwen3.8-Flash-Next-NVFP4.

This card is deliberately blunt about what was and was not established. The short version: this is a maths drafter, a code wash, and it makes chat slower at every block size we can serve. It beats the target's own tuned speculative head by 3.87% in aggregate, in eager mode, and it is a research artifact rather than a supported product. If you only read one section, read What is not established.

Companion write-up, including the four measurement mistakes that produced this artifact: Our 25% inference speedup became 3.9% after we fixed the benchmark twice.

---

What it is

It ships draft layers and the fusion projection only

embed_tokens.weight and lm_head.weight are not in this repo. DeepSpec initialises both from the target, freezes them, and never trains them, so shipping 248,320 × 2,560 target-derived rows would be a re-publication of the target's own weights.

A consumer must bind both from the target at load time. The checkpoint alone is not runnable. vLLM's target-module-sharing protocol does this for you once Qwen4ExpForConditionalGeneration is in the sharing allowlist, which is one of the overlay changes below. The binding was proved byte-identical: training used Qwen/Qwen3.8-Flash-Next@f5d08274, and the raw contiguous BF16 tensor hashes match the NVFP4 serving target's tensors exactly.

The tap, precisely

The five taps are the HC-contracted, native-width (2560) residual from each tapped layer's own GatedResidual mix, not the raw HyperConnection-wide (4 × 2560 = 10240) stream. This reuses the model's own learned contraction weights. In vLLM the aux boundary ids are the trained tap ids + 1 ([4, 16, 24, 36, 44]), each at layers[i].attn_hyper_connection.mix/combine_and_mix(...)[1].

Concatenation is [T, 5 × 2560] = [T, 12800], matching fc in_features. The final hidden state was the distillation target during training, not a sixth conditioning tap.

---

Measured serving results

Protocol. 2× NVIDIA DGX Spark (GB10), vLLM, TP2 + expert parallel, eager, c=1, temperature 0 (greedy), non-thinking, 256-token outputs, 8k context, gpu-memory-utilization 0.75. One frozen 100-prompt fixture (33 code / 33 math / 34 chat) drawn from the held-out split of the training corpus. Two boots per arm, two runs of 100 prompts per boot, except where the arm-count column says otherwise. Every arm ran on the same patched engine; the only difference between arms is the speculative config. Headline quantity is aggregate output throughput (total output tokens / total wall time); intervals are a paired stratified bootstrap over prompts, 8,000 resamples, repeats averaged within prompt.

Headline: +3.87% aggregate over the tuned native MTP baseline (k=4), 95% CI [+2.10%, +5.77%]. Block 4 is +3.26% [+1.84%, +4.79%]. Both intervals exclude zero.

Arrived here from the +36.1% / +12.0% / −13.5% figures? Those are per-workload median tok/s against native MTP k=3, from the earlier single-boot run. They reproduce from the receipts and they are reprinted in full below, next to the aggregate-vs-k=4 numbers on this page: see Per-workload medians vs native MTP k=3. Everything in the table above is a different quantity: aggregate throughput against the tuned k=4 baseline.

The baseline had to be tuned first, and it moved the headline. k=4 (50.371) beats k=3 (50.071) by 0.60%, but that difference's CI spans zero, so k=3 and k=4 are indistinguishable and the honest baseline is "≈50.2 at k=3 or k=4". k=1 and k=6 are clearly worse. Everything here is quoted against k=4 because it is the harder baseline. Against k=3 the same block-5 arm is +4.50% [+2.35%, +6.80%], which is the figure an earlier version of this card carried as +4.59% [+2.14%, +7.06%] on single-boot data.

Boot-to-boot variance is not in these intervals, and it is the same size as the effect. The paired bootstrap resamples prompts, not boots. Two boots cannot estimate a boot variance component. Observed movement between the two boots of an arm: spec off 24.304 → 25.138 (+3.4%), MTP k=4 50.876 → 49.876 (−2.0%), DFlash block 4 51.677 → 52.186 (+1.0%). Per-boot aggregates for every arm: AR 24.304 / 25.138 · MTP k=3 49.959 / 50.184 · MTP k=4 50.876 / 49.876 · MTP k=6 46.926 / 47.049 · graph 48.318 / 48.422 · block 2 46.080 / 46.150 · block 3 49.721 / 50.199 · block 4 51.677 / 52.186 · block 5 52.253 / 52.391 · block 7 50.669 / 50.403. This is the weakest part of the result. Read every interval above with it.

Two arms are thinner than the rest: native MTP k=1 is one boot (2 runs), and DFlash block 4 is 3 runs, not 4. Blocks 2 and 3 are patched arms: they required widening our own K allowlist in the serving plugin (serving/plugin/dflash_epoch7.py:68) from (4,5,7) to (2,3,4,5,7). That guard was a scope marker, not an architectural limit — blocks 2 and 3 take the identical proposer path — but the label travels with the arm.

Per workload, aggregate tok/s against native MTP k=4 on the same prompts

Say it plainly: this is a maths drafter. Code is a wash and chat is slower. At block 5 the code CI spans zero (+0.37% [−3.17, +4.21]), so we do not claim a code gain against the tuned baseline. The blended +3.87% is math carrying code and chat.

Chat regresses at every block we can serve, and every interval excludes zero: block 2 −6.13%, block 3 −3.87%, block 4 −3.90%, block 5 −5.90%, block 7 −11.32%. Chat does not keep improving as the block shrinks. It bottoms out at blocks 3 and 4 around −3.9% and then gets worse again at block 2, which also gives up most of the math gain (−7.28%). Against the softer k=3 baseline the chat regression is larger still: block 4 −7.08%, block 5 −9.01%.

The drafter only out-accepts the native head on math. Served accepted length, block 5 against MTP k=4: code 2.99 vs 3.62, math 4.24 vs 3.98, chat 2.14 vs 2.76. On chat it accepts less while paying a cheaper pass (54.3 ms vs 66.0 ms) and loses anyway.

Served accepted length above is the engine-counter quantity (1 + accepted/drafts). Cross-checked independently for block 5 from the recovered streaming block structure: 25,600 tokens in 8,889 engine steps = mean block 2.880, against the counter-derived 2.883.

Per-workload medians vs native MTP k=3, as first published

These are the three figures we posted publicly when this drafter first went out, kept here so that post is checkable against the receipts. They are not the table above, and they are not the headline. Different quantity, different baseline, different run.

Measurement definition, in full: block 5, per-workload median of per-request output throughput, on the same 100-prompt fixture (33 code / 33 math / 34 chat), on the same 2× NVIDIA DGX Spark (GB10) pair, greedy, non-thinking, 256-token outputs, eager, c=1, against native MTP k=3 — the untuned native baseline, measured on the same harness in the same session. One clean boot per arm, 2 runs × 100 prompts.

The posted one-decimal figures — 78.3 / 58.7 / 37.8 tok/s and +36.1% / +12.0% / −13.5% — are these values truncated to one decimal. All three percentages reproduce exactly from the per-request records. Note that 58.789 rounds to 58.8, not 58.7. Aggregate throughput for the same arm on the same run was 52.239 vs 49.947, +4.59%, which is the figure an earlier version of this card carried.

This run is single-boot, so the boot-to-boot variance disclosure below applies to it more strongly than to the two-boot k=4 numbers, not less.

Why the two per-workload tables differ

The k=4 table is aggregate throughput (total output tokens ÷ total wall time) against the tuned native baseline, over two boots per arm with a paired bootstrap. The k=3 table is the median of per-request throughput against the untuned k=3 baseline on a single boot. Medians discard the slow tail that aggregate throughput has to pay for, and k=3 is the softer baseline, so the same drafter reads higher under the second framing. Math and chat move modestly between the two; code is the figure that moves materially, from +12.0% to +0.37%, and under the aggregate-vs-k=4 comparison the code gain is not distinguishable from zero.

A note on the baseline

Two corrections landed on this baseline, in order. First, an earlier version of this comparison used a 41.9 tok/s MTP k=3 baseline measured on a different harness with eight short synthetic prompts; re-measuring on this fixture moved it to ≈50. Second, that re-measured baseline was still untuned: we had never swept k. Sweeping it put the best native arm at k=4. Nothing on this card compares numbers measured on different harnesses, and nothing compares against an unswept baseline.

Protocol deviation, stated

The measurement brief asked for ≥5 runs × 30 prompts per boot. We ran 2 runs × 100 prompts per boot instead. The fixture is fixed-order, so --limit 30 would have measured the same 30 prompts every run (11 code / 10 math / 9 chat), giving per-workload intervals on 9 to 11 prompts. For a bootstrap that resamples prompts and averages repeats within prompt, 100 prompts × 2 runs strictly dominates: 100 resampling units instead of 30, the full 33/33/34 strata, and 4 repeats per prompt across the two boots.

---

Losslessness — CERTIFIED for the accept path at block 5

An earlier version of this card said losslessness was not certified, and listed what a real certification would need: common-prefix controls, block-boundary recovery so that accepted drafts can be separated from the block-final correction slot, a mismatch rate broken out by position within the block, and a measured nondeterminism floor to compare against. All of that has now been run on the winning arm.

Method. One continuation was captured from the DFlash server at block 5, with engine-step block boundaries recovered from the streaming chunk boundaries, so every emitted position is labelled as either an accepted draft or the block-final bonus/correction token. That same continuation was then scored under teacher forcing on the DFlash server and twice on a spec-off (AR) server. Every arm therefore sees an identical prefix, and AR-replay-1 against AR-replay-2 gives the pure nondeterminism floor at exactly those positions.

On the positions the accept path is responsible for, the drafter disagrees with the spec-off target less often than the target disagrees with itself (0.784% vs 0.844%). SE ≈ 0.07 pp, so this check resolves to roughly 0.2 pp; the previous version was blind below about 2.5%.

Mismatch by position within the block. A real accept-path bug concentrates at one offset; noise does not:

No concentration anywhere. Violation tracks the floor at every offset and declines monotonically.

The ">12 log unit" mismatches are explained and the worry is retired. 49 violations exceed 4 log units, maximum 12.0, and all 49 sit at the block-final position — the bonus and correction slot — with none at an accepted-draft position. The same tail appears when the DFlash server scores its own capture (53 such cases, maximum 12.9), so it is a property of teacher-forced replay at the correction slot, not of the accept path. The earlier claim "all mismatches are near-ties" was false; the correct statement is that the large-margin mismatches live at the slot the accept path does not own.

Scope, so this is not over-read. This certifies the accept path at block 5, greedy, under teacher forcing, against a measured nondeterminism floor on this stack. It is not an absolute losslessness claim, and it says nothing about sampling, other block sizes, or long context.

---

Graph mode — memory fits, graphs hurt the baseline, and our own adapter blocks the drafter

An earlier version of this card said graph memory fit had never been demonstrated and that eager mode might be flattering the drafter. Both statements are now superseded by measurement.

• Graph memory fits under the 0.75 policy. This is the first demonstration. Native MTP k=4 booted twice with {"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8,16]} at gpu-memory-utilization 0.75 and served 4 full runs. The earlier attempt died inside profile_cudagraph_memory → initialize_kv_cache, before capture.
• Graphs make the baseline slower, not faster: 48.370 tok/s graph vs 50.371 eager, −3.97% [−4.68%, −3.26%], consistent across both boots and all three workloads. So the eager-only protocol was not flattering the drafter. The earlier speculation that eager inflates per-pass overhead and therefore favours a one-parallel-pass drafter is not supported; if anything, graphs would widen the drafter's margin.
• The DFlash half of the A/B cannot be run, and the blocker is ours. It fails at serving/plugin/dflash_epoch7.py:80 with ValueError("HC attachment requires enforce_eager"), raised in the draft model's __init__ during load. That is our own adapter's assertion, written as a correctness guard on the HC tap. It is not a vLLM limitation and not a memory failure. The graph comparison is therefore one-sided, baseline only, and removing that guard is our work to do.

---

The engine is not run-to-run reproducible at c=1, temperature 0

This is unflattering to the whole setup and it is the honest resolution floor for every check on this card, so it is stated here rather than buried.

Measured same-boot, free-running, on identical prompts, with a common-prefix estimator (the rate at which two runs first disagree, given they have agreed so far):

At roughly 2% per step, almost every 256-token greedy generation differs somewhere between two runs of the same server: eight of the eleven arms diverge on 98% or more of requests, and the three best-behaved still diverge on about 81%. Batch size one, temperature 0, same server, same prompts.

The consequence: any gate written on free-running output identity cannot pass, including the harness's own token_agreement ≥ 0.99. Not at any threshold above the floor, for any configuration, drafter or not. A ~2.7% figure carried in earlier notes was a teacher-forced per-position rate, not this one. If you are building acceptance gates for speculative decoding, measure your engine against itself first.

---

What is not established

This section is not a footnote. Every item below is a thing we did not do, and each one limits what the numbers above mean.

Boot-to-boot variance is not in any interval on this card, and it moves as much as the effect. Two boots per arm cannot estimate a boot variance component. Observed movement is +3.4% / −2.0% / +1.0% on three arms, against a headline effect of +3.87%. Per-boot aggregates are published above so this can be read directly. This is the single weakest part of the result.

No release gates were run. No concurrency (c=8, c=16), no soak, no thinking-mode measurement, no sampling measurement, no long-context measurement. Everything is greedy, non-thinking, 256-token, 8k context, ignore_eos. The math gain is unmeasured in the mode a user chasing a math accelerator on this hardware would actually run, which is thinking mode with sampling.

The graph-mode comparison is one-sided. Baseline only, because of our own adapter's enforce_eager assertion. Graph-mode throughput for the drafter is unmeasured.

Two arms are thin, and two are patched. Native MTP k=1 is one boot; DFlash block 4 is 3 runs rather than 4. DFlash blocks 2 and 3 required a one-line widening of our own K allowlist and are labelled patched wherever they appear.

Losslessness is certified only for the accept path at block 5, greedy, against a measured floor. Not for sampling, not for other blocks, not for long context, and not absolutely.

An earlier release bar of ≥25% aggregate uplift was not met and was subsequently withdrawn as arbitrary. We are recording that rather than quietly dropping it. Changing the objective after seeing results is exactly the failure mode this card exists to avoid.

---

The offline metric caveat, and why you should care even if you never use this drafter

Our training curve used a proxy: the product of marginal per-position agreement rates. That is not the probability of an accepted prefix, and it is wrong twice over. The per-position rates were averaged over every anchor still valid at that position rather than every anchor still alive, so drafts that had already diverged stayed in the denominator and could score correct later. And multiplying marginals assumes independence across positions inside one block, which is false, because those positions share a context and are strongly positively correlated. The two errors point in opposite directions.

The corrected metric walks each block in order and counts position k only if every earlier position in that same block was accepted, reporting E[L] = 1 + Σ_k P(prefix ≥ k), which is what the server's own counter measures. The corrected evaluator and its 13 unit tests are published in the serving repo.

The corrected number does not predict served accepted length. It is worse at it than the broken one was, and the error grows with block length:

The corrected offline hazard rate is nearly flat with depth (0.815, 0.800, 0.813, 0.828, 0.831, 0.828, 0.813). Served acceptance collapses with depth. That is a disagreement about shape, not a level offset, so no rescaling closes it.

Leading hypothesis, untested: offline evaluation samples anchor positions uniformly over the corpus, while a server re-anchors precisely where the previous block was rejected, and so disproportionately starts drafting at hard positions. If that holds it is structural to speculative decoding rather than a bug, and it would make uniform-anchor offline evaluations systematically optimistic. We have not run the experiment. Two other candidates we have not ruled out: the served hidden-state tap may differ from the trained tap, because the engine patch permits a fallback buffer; and the offline and served workloads are different data.

Do not quote any offline number on this card as expected serving speed. That includes 4.3371. This is a warning to anyone evaluating drafters offline, including us.

Retired for the record: τ = 3.079. It is neither the marginal curve nor an accepted length, and any DFlash-versus-DSpark comparison made on the marginal curve was structurally biased and should be discarded.

---

Serving requirements

Stock vLLM cannot load this drafter. You need three control-flow patches and a six-file adapter overlay. The patches alone serve nothing. All of it is published as diffs, with base and patched sha256 for every file, at PixelML/deepspec-qwen38-flash-next.

Pinned engine: vLLM source e962733e08d10f7ca65dac4df99e116460b8b174, image vllm/vllm-openai@sha256:89dd8f442a3f4c08c6b3cd634c4f735cd709160651c296596673cf974ea6ee39 (arm64).

The three patches, and the failure each one fixes

There is no V1 fallback: under VLLM_USE_V2_MODEL_RUNNER=0 this target raises RuntimeError: PLE inputs were not prepared and does not boot at all, drafter or no drafter.

The six-file adapter overlay

It makes the target emit contracted auxiliary hidden states at all, and teaches the proposer DeepSpec's anchor convention: aux capture at the HC boundary layers in vllm/models/qwen4_exp/nvidia/model.py; query_zero_predicts_next in vllm/v1/spec_decode/dflash.py; the matching sample_offset in the Triton kernel in vllm/v1/spec_decode/utils.py; the target-sharing registration in vllm/v1/spec_decode/llm_base_proposer.py (this is what binds the embedding and LM head); one scheduler-slot adjustment in vllm/config/speculative.py; and the dflash allowlist entry in vllm/model_executor/models/config.py.

You also need the drafter config override in the serving repo, because this repo's config.json does not carry the dflash_config block the engine reads.

Filed upstream as vllm-project/vllm#56088.

SGLang cannot serve this class of drafter on this target at all. Filed upstream as sgl-project/sglang#38589. Note that where vLLM raises on the anchor layout, SGLang would shift every draft position by one silently.

---

Exporter fidelity

Features were exported from vLLM, not transformers, because vLLM is the engine that generated the training corpus and the engine this drafter is served against. Fidelity against a BF16 transformers reference, 20 prompts, 9,214 tokens:

lm_head(last_hidden) reproduces the engine's own top-1 on 0.99695 of 9,194 positions.

The acceptance gate for this build was "per-tap cosine ≥ 0.98", and the deepest taps came in at 0.9761. The build proceeded anyway, as a labelled deviation: the comparison pits NVFP4 against BF16 and here the BF16 path is the approximation, the gap decays monotonically with depth (the compounding signature of quantisation rather than a wiring defect), and the internal lm_head check that quantisation cannot affect passes at 0.99695. That reasoning is an argument, not a proof, and it is recorded here as one.

Separately, a cross-platform teacher-fidelity gate set at 98% token agreement measured 97.478% and was replaced, because the same engine agreed with itself on a bit-identical rerun only 97.283% of the time. The corrected gate — cross-platform disagreement must not exceed same-platform run-to-run disagreement — passes, but on ~4,600 positions the standard error is about 0.24 percentage points and the two numbers are 0.2 points apart. That pass is a coin flip on point estimates, not a demonstration. See the engine is not run-to-run reproducible for the properly measured floor this gate was sitting under.

---

Training

DeepSpec's own reference recipe is roughly 1.42M samples, ~1.2B tokens, 10 epochs. We ran about 7% of the samples, 10% of the tokens, and 7 epochs. That was a deliberate, priced deviation. On our measurements accepted length was not obviously the binding constraint — block 7 served an accepted length of 2.984, within noise of native MTP k=3's 2.997, and was still slower than the drafter's own block 5 on the wall clock — but the tuned baseline at k=4 accepts 3.367, which the drafter does not match at any block, so we cannot claim more data would not have helped. We also cannot reason about it with the metric we used at the time, because that metric is the retired proxy.

The training prompt mix follows DeepSpec's, which is about 78% maths and code. We then evaluated on a balanced fixture and found chat regressed. Those two facts are probably related; we did not run the ablation that would show it.

---

Licence and attribution

Built with Qwen.

• Code. DeepSpec and DFlash are MIT. Their LICENSE and NOTICE files travel with the source repo. DFlash: z-lab/dflash, MIT. DeepSpec: deepseek-ai/DeepSpec, MIT.
• This artifact. It was trained entirely on outputs of Qwen3.8-Flash-Next, served through NVIDIA's NVFP4 build of it. The Qwen Community License 1.0 and the NVIDIA Open Model License therefore apply to use of these weights, in addition to the MIT terms on the code that produced them. MIT on the trainer does not resolve the obligations attached to training on a model's outputs. Read both licences before deploying.
• No target weights are redistributed here. The embedding and LM head are stripped and bound from the target at load time.
• The training corpus is not published. The prompts are Apache-2.0 / MIT open sources, but the responses are Qwen model outputs and carry the licence terms above. We are not redistributing another model's generations at scale.

---

Related artifacts

• Serving repo, patches, exporter, corrected evaluator: PixelML/deepspec-qwen38-flash-next
• Write-up: the blog post
• vLLM upstream issue: vllm-project/vllm#56088
• SGLang upstream issue: sgl-project/sglang#38589

A DSpark variant of this drafter was also trained and is not published. It serves at −11.0% against native MTP k=3 and roughly doubles the chat regression, and its per-request confidence gate cannot run on this target at all: vLLM's adaptive verification needs to trim verification requests on device, and GDNAttentionBackend does not support that. Since Qwen3.8-Flash-Next has 36 GatedDeltaNet layers, that check can never pass. This generalises to every Mamba-class hybrid target.