DeepSeek-V4-Flash-DSpark-draft-GGUF

DSpark drafter for DeepSeek-V4-Flash — GGUF (llama.cpp)

First working DSpark speculative decoding for DeepSeek-V4-Flash under llama.cpp. This repo ships the drafter as a single GGUF (DSV4-Flash-DSpark-draft-bf16.gguf, 10.9 GB, arch dflash, block size 5) plus the port that runs it. The target model is untouched: any existing DeepSeek-V4-Flash GGUF (e.g. Unsloth's quants) works as-is, no reconversion.

DSpark (arXiv:2607.05147) is DeepSeek's block-parallel drafter: a small semi-autoregressive model with a DeepSeek-V4 backbone (MLA + MoE + hyper-connections) that drafts 5-token blocks from the target's own hidden states and shares the target's embeddings and lm_head. DeepSeek ships its weights inside the DeepSeek-V4-Flash checkpoint (mtp.{0,1,2}.* tensors), but no runtime outside their internal stack supported it. This port makes it run in llama.cpp.

Requirements

DSpark support for DeepSeek-V4 targets is not yet in llama.cpp master. Build from this branch (stacked on ggml-org/llama.cpp#25173, which introduces the dflash architecture with a Qwen3-style drafter):

git clone https://github.com/YanissAmz/llama.cpp -b dspark-dsv4
cmake -B build -DGGML_VULKAN=ON   # or your backend
cmake --build build -j -t llama-server

The work was submitted upstream as ggml-org/llama.cpp #25683 (the port), #25682 (a fix to the server's draft-acceptance statistics), and #25687 (batched device reads on state save, ~12× faster checkpoints — model-agnostic). A follow-up branch dsv4-partial-rollback adds bounded partial rollback to the DSV4 KV cache, which removes the per-step checkpoint and the replay decode entirely; its numbers are included below.

Run

./build/bin/llama-server \
  -m DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf \
  -md DSV4-Flash-DSpark-draft-bf16.gguf \
  --spec-type draft-dspark \
  -ngl 99 -fa on --jinja

--spec-type draft-dspark is required. draft-dflash selects the denoising path of #25173 and silently produces a draft shifted by one position.

To reconvert the drafter yourself from the official checkpoint (only shards 46–48 of the HF repo carry the mtp.* tensors): the converter routes to the drafter automatically from the flat dspark_* hparams in config.json — same DeepseekV4ForCausalLM architecture string as the target, no flag needed. It only needs the target's directory for the shared tokenizer:

python convert_hf_to_gguf.py <drafter-shards-dir> \
  --target-model-dir <DeepSeek-V4-Flash-dir> --outtype bf16

Acceptance

DeepSeek-V4-Flash UD-IQ3_XXS target, block size 5, greedy, Vulkan (AMD Strix Halo APU):

6.00 is the ceiling for a 5-token block (block + bonus token). The per-position acceptance stays flat across the block, which is the signature the DSpark paper reports (its Figure 2 contrasts this with the rapid decay of a plain block-diffusion drafter). Getting the flat curve required finding a subtle bug: LLM_ARCH_DFLASH rotated its rotary dimensions the NeoX way, correct for the Qwen3-style drafter of #25173 but wrong for a DeepSeek-V4 backbone, which pairs them interleaved (NORM, like deepseek2/deepseek4). A drafter has to rotate exactly like the target it drafts for. The failure mode is nasty because nothing crashes: position 0 of every block stays correct (carried by the injected target hidden state and the shared lm_head) while deeper positions silently lose their positional signal and repeat earlier ones — acceptance just stalls at ~2 tokens.

Throughput (Vulkan, short context)

Same setup, greedy so both configurations generate identical tokens. Baseline without speculation: 12.53 t/s. Scope: Vulkan backend, short prompts, and a build without the decode work described in "At long context" below. Every number in this section is that configuration; see the next section for what happens at 176k on a faster target.

Two systemic costs had to fall before the drafter paid for itself on this hardware, and both fixes are model-agnostic:

1. State checkpoints were 12× slower than necessary (#25687). The DSV4 cache refuses partial rollback, so the server checkpoints the whole sequence state before every draft. The checkpoint copied 11.6 MiB in 80 ms — not because of size, but because it issued one synchronous device→host read per tensor (~120 submit+fence round-trips on Vulkan). Batching them through async reads plus a single synchronize brings it to 6.4 ms.
2. Rejected drafts paid a checkpoint-restore plus a replay decode (dsv4-partial-rollback). The DSV4 compressed caches are ring-addressed (pos % state_size), which is why rollback was refused wholesale — but the rings are sliding windows, so oversizing them by the draft length makes exactly the last few positions removable, and the server skips both the checkpoint and the replay. At 0.72 acceptance the target used to run 22% slower with speculation than without (9.75 vs 12.53); that collapses to ~3% slower (12.15 vs 12.53).

Read that last row honestly: 3% slower is not a win. The open-ended code prompt is still, marginally, better off with speculation switched off — the rollback fix turns a catastrophic regime into roughly a wash, and 3% is only about 2× my measurement noise, so "wash" is the fair word rather than either "fixed" or "loses badly". Speculation pays here on the repetitive prompts (+19% and +25%), where acceptance is 0.95-1.00.

Honest caveat: these are APU numbers, where an 11 GB drafter is expensive relative to the target's decode. On hardware where the drafter's forward pass is cheap relative to a 284B target, the acceptance rates above imply a substantially larger win — but I have not measured that (my only discrete GPU is out of action), and I'd rather say so than imply a speedup I did not observe.

At long context on this APU (AMD Strix Halo, ROCm): the drafter does not pay

Scope, before anything else. Everything in this section was measured on one machine — an AMD Strix Halo APU under ROCm. Two different kinds of numbers follow, and they travel differently. The acceptance numbers (which tokens the drafter predicts correctly) are a property of the drafter–target pair, not the hardware; if you run DSpark on NVIDIA, they should transfer, and the regime × depth table below is the part written for you. The decode/step-cost numbers — and the "does not pay" verdict built on them — are this APU's economics: a full drafter forward here costs a large fraction of a target step. On a discrete GPU where the drafter's forward is cheap relative to the target, plug the acceptance table into acc = (raw·step − 1) / 5 with your step costs; the verdict can invert, and nothing below claims otherwise.

Since publishing the numbers above I spent a campaign optimizing DSV4-Flash decode on the same APU under ROCm/HIP. Raw decode at 176k context went 7.56 → 13.55 t/s (+79%), quality-gated (wikitext PPL 4.0902 vs 4.0926, −0.059%), from four changes that compose: a GPU top-k for HIP (it was running on the CPU), a KV co-allocation, flash-attention mask compaction, and a templated hyper-connection kernel.

That makes this drafter's case worse, not better — and it is worth stating plainly, because it is the opposite of what the caveat above hopes for. The caveat says the drafter is expensive relative to the target's decode. Speeding the target up by 79% while the drafter's forward pass costs exactly what it did before makes the drafter relatively more expensive. The faster the target, the more acceptance you need before drafting pays.

All numbers below are 176k of real KV, greedy, n_max=5, measured on one binary (the drafter is merged into the optimized tree, so the target runs the full stack). Restored from a saved slot, prompt_n verified on every sample so the depth is real and not a silent re-prefill.

(The raw comparator here reads 13.32–13.41, not the 13.55 quoted above. Both are raw decode at 176k on the full stack; they are different sessions, ~1.4% apart, which is this box's run-to-run spread. I quote the same-session pair rather than the headline number, because the only comparison that means anything is the one where a single variable moved.)

DSpark is a net loss at 176k on this hardware. Code is what this drafter is good at, and writing a new kernel still lands at 0.70× of simply not speculating. And that is not for lack of trying easier tasks: I also measured the easiest one that exists for a drafter — verbatim recall of a function whose full text sits ~107k tokens upstream in the same context. The target reproduced it 100% byte-identical on every sample, the drafter posted its best acceptance ever (0.880–0.921) — and decode was 12.73–13.32 against raw's 13.32–13.41: three narrow losses and one tie at raw's lower bound. There is no measured regime, including perfect recall, where -md beats raw at this depth on this hardware.

Why: at this depth a drafting step costs ~380–425 ms nearly regardless of acceptance — 379 ms at acceptance 0.29, 385 ms at 0.39, 391 ms at 0.54, ~423 ms at 0.88–0.92. The work per step is fixed to first order (five drafter forwards, plus one 5-wide verify over the same 176k KV); acceptance only changes how many of those tokens you keep. There is a real second-order drift — the step rises ~12% across that acceptance range, monotonically, which is why the highest acceptance doesn't clear raw even when a fixed-step calculation says it should; my candidate mechanism is the drafter's catch-up batch growing with accepted length, and I did not isolate it. So decode ≈ (1 + 5·acc) / step, and beating raw 13.4 needs acceptance somewhere in 0.81–0.94 — a range, not a point, because it inherits the step spread and I would rather show you that than round it away. The drafter delivers 0.54 on code, which is under every estimate in the range. The verdict doesn't depend on where in it the true threshold sits, and the gap is not something a kernel closes.

Depth is part of it, but only for code:

Depth costs invented code acceptance about a quarter of its value (0.72–0.75 → 0.54) and leaves prose alone. That first row is the one to read for the depth effect: same invented kernel, same stub byte-for-byte, and the short context is a strict prefix of the long one, so depth is the only variable that moves. The recall row bounds what depth is not doing: it doesn't cap acceptance at 176k — the drafter's highest acceptance anywhere in this project was posted at depth, on recall. Depth hurts the drafter specifically where the continuation is unconstrained by nearby text. The drafter's advantage on code over prose — around 3–4× at short context — compresses to around 2× at depth. Those are rough magnitudes on purpose: they are ratios of three-sample means whose own ranges are ~20% wide, and the honest content is the direction and the non-overlap, not a third significant figure. The interaction is the interesting part and I have no mechanism for it, only the measurement.

A correction I'd rather show than bury: an earlier draft of this section said depth cost code acceptance a third, and collapsed the two rows above into a single row labelled "code". It doesn't, and there is no single such row. That number compared inventing a novel function at 176k against continuing a truncated file at short context — two different generation tasks wearing one label — and charged the whole gap to depth. (An earlier fix of this paragraph called that short cell "reciting". Also wrong, caught the same way — by a measurement refusing to match the label: the truncated file's remainder is not in the prompt at all, and true recital, measured separately, posts 0.88–0.92, not 0.78–0.83.) The continuation-over-invention edge looks worth something like 4–14% of the acceptance — though that leg is the soft one: the two short-context cells read different source files, so it measures regime plus text, not regime alone. The fix was the middle row: run the same invention task at short context. Depth survived at a smaller size; the missing leg was real. The tell was in my own table before I ran anything — both prose cells were continuations and showed no depth effect, and the only row that moved was the only row whose two ends were different jobs. If a 2×2 shows an interaction on exactly one row, doubt that row's noun before you believe the interaction.

The sensitivity check became a measurement. An earlier draft of this paragraph granted the drafter its best short-context acceptance at depth by assumption and derived a narrow loss (12.5–13.2) from step costs. The assumption is no longer needed: the recall row is the best case, measured at depth. Acceptance 0.880–0.921 — the highest this drafter has posted anywhere, on the easiest task that exists for it, with the target's output 100% byte-identical to the in-context original on every sample. Decode: 12.73 / 12.83 / 12.87 / 13.32 against raw 13.32–13.41. Three narrow losses and a tie at raw's lower bound. The derived numbers were honest arithmetic and landed where measurement landed; they're superseded anyway. Note what the measurement adds that the arithmetic couldn't: at a fixed 391 ms step, acceptance 0.92 should clear raw at ~14.3 — it doesn't, because the step itself drifts up to ~423 ms at high acceptance. The break-even recedes as acceptance approaches it: at that step it sits at ~0.93, above everything measured. Which pins the true threshold near the top of the published 0.81–0.94 range, and closes the question: there is no acceptance this drafter has ever posted — at any depth, on any task, including perfect verbatim recall — that makes -md beat raw at 176k on this hardware.

Two honesty notes on method. (1) The raw figure is not a no-draft arm inside the same session — there cannot be one, since the drafter is attached at server start and every request in a DSpark session drafts. The comparison is cross-session, on the same binary, and the two raw points span 0.63%, which the 1.4× gap dwarfs. The DSpark arm also carries an 11 GB resident drafter and rings sized for the draft block; both are costs of turning DSpark on, so charging them to DSpark is the question a user actually faces ("should I pass -md?"), not a confound. (2) Acceptance is a property of what the model generates, not of what is in the prompt. My first attempt at the code cell fed a 176k source tree ending in an instruction, and the model answered it in English prose inside // comments — that measured a refusal, at 0.33, a prose number. The row above appends a mid-expression stub instead, so the only sane continuation is C++, and I read the emitted text on every sample to confirm it. If you benchmark a drafter, read what it actually emitted before you label the row — and then check that the label still means the same thing in the next cell over, which is the mistake documented above this one.

One note on that break-even, since an earlier draft of this section said I had none to quote. I had a threshold in my notes whose derivation did not survive review, and it is gone. The 0.81–0.94 above is not a rehabilitation of it — it is a different quantity, derived only from the step costs measured on this page, and you can rebuild it from the numbers given: acc = (raw·step − 1) / 5.

This does not mean DSpark is a bad drafter — the flat per-position acceptance curve above is real and is the paper's signature. It means a ~11 GB drafter running a full forward pass per block does not pay against a 284B target whose decode is already fast on this APU, at a depth where the verify step is expensive and acceptance falls on exactly the novel code you'd want a drafter for — even the 0.92-acceptance recall regime only reaches a tie. On a discrete GPU, where the drafter's forward is cheap relative to the target, the conclusion could easily invert; I still cannot measure that.

The decode work above lives in a separate fork branch and is not part of this repo:

• Fork branch: YanissAmz/llama.cpp feature/dsv4-strix-halo-decode
• It builds on ggml-org/llama.cpp#25545 (@fairydreaming's lightning-indexer kernel). Not upstreamed.

Serving config (Strix Halo, 128 GB): launcher, sweeps, and where it does pay

The section above measures this drafter at 176k against the text DeepSeek-V4-Flash and concludes it does not pay. That verdict stands for that configuration. It is not the whole picture, and this is the other half.

Scope: ROCm/HIP, a different target — DeepSeek-V4-Flash-Vision-Exp (deepseek4, UD-IQ3_XXS, ~97 GB), n_max=3, p_min=0.20. Not the Vulkan setup of "Throughput (Vulkan, short context)" above, which is why the absolute numbers are far higher; the two short-context sections are not comparable to each other. Drafter compatibility was checked on the metadata first: same vocab (129280), 3 blocks, 81 tensors.

On DeepSeek-V4-Flash-Vision-Exp at short context (~275 tokens, window 131072), the same drafter takes decode from 15.57 → 24.04 t/s (+54.4%) — paired duel, two alternating legs, eviction delta 0. Acceptance 0.609 aggregated over 5952 drafted tokens; mean accepted length 2.81.

Both results follow the same rule, and the arithmetic is decode = mean_len / step:

The two targets decode at a comparable rate without a drafter (64 vs 75 ms/token). What differs is the drafting overhead — 53 ms against 316 ms — and the section above already names the reason: the faster the target and the deeper the context, the more acceptance you need before drafting pays. At 275 tokens the bar is low. At 176k nothing this drafter has posted clears it.

Full launcher, the n_max and p_min sweeps (n_max=3, p_min=0.20 ships), the paired-duel harness, and three other models on the same box:

https://github.com/YanissAmz/strix-halo-llm-serving

Notes

• The drafter's RoPE carries no YaRN, on purpose: the reference implementation disables YaRN on pure sliding-window layers, and every layer of this drafter is one.
• Greedy speculative output can legitimately differ from greedy non-speculative output on near-ties: the target evaluates a draft as one batch, which reorders float reductions. This happens on master today, with or without this port (ggml-org/llama.cpp#25618). The rollback correctness gate used instead is exact: decode a wrong tail, roll it back, replay the true tail, and require max logit diff = 0 against a context that never rolled back — with the rolled-back span placed to straddle an HCA block boundary (ratio 128), which a short test would never exercise.
• Fixes to #25173's shared path that also matter for the Qwen drafter: the draft sampler discarded its sample and re-read cur_p->data[0] (forcing greedy); the block's noise K/V from the previous iteration was left in the drafter's cache (the block attends non-causally, so the next block saw it); target hidden states were injected over an unpurged draft region.

Related

• YanissAmz/strix-halo-llm-serving — launchers, drafter sweeps and paired benchmarks for four large models on this box
• YanissAmz/Hy3-295B-A21B-GGUF — Hunyuan3 with MTP speculative decoding under llama.cpp (+40% decode)
• YanissAmz/Nemotron-3-Puzzle-75B-A9B-GGUF — NVIDIA Puzzle port with MTP