Qwen3.5-0.8B-DSpark

Qwen3.5-0.8B DSpark drafter (v0.3, experimental)

Card status (2026-07-25, corrected): all numbers below are measured. The matched-vs-generic A/B has landed (see its section) and the vLLM smoke test passed (see How to run). 2026-07-31: vLLM acceptance fixed and verified at llama.cpp parity (0.155 vs 0.158 per drafted token). The earlier vLLM shortfall was a config.json rope bug in this repo (partial_rotary_factor inherited from the target), not a runtime difference; the shipped config now sets it to 1.0 explicitly. See the vLLM and cross-stack sections. Correction: an earlier version of this card described a "quant tax" and claimed a bf16-trained head loses most of its acceptance on aggressive quants. Follow-up work refuted that: acceptance tracks whether the target is still coherent, not the precision gap between head and target. The affected sections are rewritten below, with the evidence.

A DSpark speculative-decoding draft head for Qwen/Qwen3.5-0.8B, trained from scratch with the DeepSpec toolkit via our fork branch satindergrewal/DeepSpec:qwen35-lane (qwen3.5-family target support, online-training mode merged from PR #23 by Ofir408, warm-start hook). As far as we know this is the first published DSpark head for a vanilla Qwen3.5 model (see DeepSpec issue #22, an open request for exactly these heads).

Status: v0.3 released. Use head_q08_v03.gguf. v0.3 doubles the self-distilled corpus (27,576 samples, 2.1x v0.2) with the recipe otherwise pinned, including the batch config the v0.2 caveat below promised to pin (global batch 128, matching v0.2). Acceptance improves monotonically in every measured cell; see the v0.3 section. v0.1 and v0.2 remain in the repo for reproducibility.

Previous status: v0.2 released, head_q08_v02.gguf. v0.2 keeps v0.1's recipe, size, and sample count and trains on far richer self-distilled data: full-length target responses instead of truncated ones, roughly 9x the supervised tokens. That bought +30-49% relative acceptance (see the v0.2 section below). v0.1 remains in the repo for reproducibility, and the measurement sections below (economics ladder, target-quant ladder, per-dataset eval) were measured on v0.1 unless labelled otherwise; the relative structure of those ladders is head-version-independent.

v0.3: the supervision-volume line is now three points, still rising

v0.3 changes exactly one variable against v0.2: corpus size. Same generation pipeline (responses regenerated by the target itself), same 10 epochs, same global batch 128 (the batch confound v0.2 flagged is pinned this time), same everything else. 27,576 kept samples against 13,118.

Gate: identical to the v0.2 gate. llama.cpp --spec-type draft-dspark --spec-draft-n-max 7, greedy, 200-token completions, two fixed prompts (P1 code, P2 counting), bf16 and Q8_0 targets, hash over reasoning_content plus content (both fields; content alone is often empty on this reasoning model).

• Monotone in every cell, v0.1 to v0.2 to v0.3. P1 acceptance is +50% relative over v0.2.
• Output identity across heads: under greedy decoding all three heads produce the same target-verified completion per (target, prompt); the output hash is identical across v0.1/v0.2/v0.3. Speculative decoding is lossless when the target verifies. The head changes how much draft is accepted, never the text.
• No plateau at 28k. The next volume step remains the cheapest lever before architecture changes:

Training: 2,150 steps (5 epochs at 27.5k samples), final loss about 1.06 from 2.8. The whole train-export-gate sequence ran unattended: a watcher keyed on the trainer PID exiting plus a size-stable final checkpoint fired the export and the three-head gate with no human in the loop.

Drafter-side quantization is verification-lossless, now shown at 744B scale

The D1 claim this card previously held back is now measured at scale: quantizing the MTP drafter block of a 744B-class MoE (GLM-5.2, blk.78 experts to q4_K) reclaimed 2.32 GiB of VRAM with byte-identical outputs and bit-identical acceptance rates across three independent runs. Verification makes drafter-side precision a pure throughput knob, from 0.8B heads to 744B internal drafters.

v0.2: supervision volume, measured (with a corrected attribution)

Correction (2026-07-28): the first version of this section claimed v0.1 trained on generic dataset responses and v0.2 on target-regenerated ones, framing this as a distribution-matching experiment. That was wrong: both versions trained on self-distilled data (responses regenerated by the target itself, same public prompt pool, same temperature). The real difference, found by auditing the training files: v0.1's regeneration capped responses at ~110 tokens on average (426 chars), while v0.2's ran to full length (~3,500 chars avg). v0.2 saw ~9x the supervised tokens, almost entirely from response length. The acceptance gains are real; the correct attribution is supervision volume (and long-form responses), not distribution matching.

Controls that held: identical architecture, block size, cold-start, epoch count (10 passes), and prompt source; near-identical sample count (13.1K vs 12.0K). Variables that differed: per-response length (~9x supervised tokens), a fresh prompt draw, and the global batch (documented below). Every training response was regenerated by the bf16 target itself (temperature 0.7, thinking disabled) instead of coming from the generic open-perfectblend pairs.

(Training-loss curves are epoch-normalized because the step counts differ, and the two losses are computed on different data, so the gap between curves is not itself a quality claim; the acceptance table above is the comparison that counts. Final losses: 1.64 vs 1.26.)

Notes for anyone reproducing:

• The v0.2 head also drafts fewer tokens while landing more (626 drafted vs 759 on the code probe): it wastes fewer draft passes, which is where the throughput gain comes from.
• Integrity anchor: in the same session, v0.1 reproduced its published CUDA reference byte-exactly (0.11594, 88/759: cross-backend greedy determinism), so these deltas sit on a zero-noise floor.
• Scope honestly: absolute acceptance stays low (0.17-0.20) because a 0.8B target is the hardest case for speculation (see the loop-rate and economics sections). This experiment proves the method at constant cost; the same recipe on larger targets is where the absolute numbers get interesting.
• Mechanism note, post-correction: this result is about how much supervised signal the drafter sees per sample (full-length responses vs truncated ones), on top of already-matched self-distilled data. Related but distinct mechanisms measured elsewhere in this project: own-data imatrix calibration helps target quants (27B card), and real importance data made no difference to MTP-block quantization at q2_K on a 27B. Measure each axis; do not assume one story covers them all.
• Control caveat, stated rather than buried: the two runs differ in one secondary knob. A config drift halved the global batch between runs (v0.1: 256, 450 optimizer steps; v0.2: 128, 1,020 steps), so v0.2 took more, smaller optimizer steps over the same 10 epochs. The data change is the dominant variable, but the batch change is a real confound this experiment did not isolate; v0.3 pinned the batch config (global batch 128, same as v0.2) and the gain held.
• The training data is published: satgeze/Qwen3.5-0.8B-selfdistill carries both versions' pairs with full provenance (prompts: open-perfectblend, Apache-2.0; responses: generated by the target, unique to that dataset).
• Training data: 13,273 samples regenerated via llama-server (bf16, 16 parallel slots) using the fork's generate_train_data.py --resume; features captured with the GGUF capture pipeline documented in the training guide linked below.

How to run it

llama.cpp (tested, all numbers on this card were measured with this)

DSpark runtime support is in PR #25173 plus this model family's support on our branch. Until both are in mainline, build the branch:

git clone --branch dspark-qwen35 https://github.com/satindergrewal/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # Metal is the default on Apple Silicon
cmake --build build -j --target llama-server

Get the head and a target:

hf download satgeze/Qwen3.5-0.8B-DSpark Qwen3.5-0.8B-DSpark.gguf --local-dir .
hf download unsloth/Qwen3.5-0.8B-GGUF Qwen3.5-0.8B-BF16.gguf --local-dir .

Serve with speculative decoding:

./build/bin/llama-server \
  -m Qwen3.5-0.8B-BF16.gguf \
  -md Qwen3.5-0.8B-DSpark.gguf \
  --spec-type draft-dspark --spec-draft-n-max 7 \
  -c 4096 -ngl 99 -fa on --jinja --port 8080

--spec-draft-n-max 7 matters: llama.cpp defaults to drafting 3 tokens, but this head was trained for a block of 7, so the default discards part of the (already marginal) speedup. Set it to the head's block size, printed at startup as block_size=.

Any OpenAI-compatible client works against http://localhost:8080/v1; the server log prints draft acceptance per request. Note the economics section below before deploying this pair: on fast hardware this 0.8B target outruns its own drafter.

vLLM (verified: full acceptance)

vLLM has DSpark support in mainline since #46995 (merged 2026-07-01, DeepSpec-format heads) and #47093 (merged 2026-07-02, speculators-format checkpoints). Requires vLLM >= 0.25 (0.24.0 predates the merges and rejects "method": "dspark"). This repo's safetensors are DeepSpec-format with architectures: ["Qwen3DSparkModel"], the exact key vLLM's registry maps for DSpark drafters.

Verified 2026-07-31 on vLLM 0.25.1 with this head (greedy, 200-token code completion): 0.155 acceptance per drafted token, position-0 acceptance 0.55, matching the llama.cpp numbers on the same probe. The mrope/rope code path is byte-identical between 0.25.1 and 0.26.x, so this carries over.

Config note (2026-07-31 fix): an earlier revision of this repo's config.json inherited the target's rope geometry (partial_rotary_factor: 0.25 plus mrope keys) while the head was trained with plain full-dimension RoPE; vLLM rotated only a quarter of each head and acceptance collapsed (position 0 ~0.26, later positions ~0). The shipped config now sets partial_rotary_factor: 1.0 explicitly; deleting the key is not enough, because the qwen3_5_text config class re-injects the 0.25 family default at parse time. If you downloaded config.json before 2026-07-31, re-download it.

vllm serve Qwen/Qwen3.5-0.8B \
  --speculative-config '{"method": "dspark", "model": "satgeze/Qwen3.5-0.8B-DSpark", "num_speculative_tokens": 7}'

Setup notes from our smoke test (RTX Pro 6000 Blackwell, sm_120a): flashinfer's device detection can fail in the engine subprocess and raise a misleading "requires sm75 or higher"; fix with FLASHINFER_CUDA_ARCH_LIST="12.0a". First run JIT-compiles kernels (needs ninja and nvcc on PATH). With a 248K-vocab model, default warmup can OOM even on 96GB; lower --gpu-memory-utilization / --max-num-seqs if it does.

SGLang

DSpark support is on SGLang's roadmap: watch the tracking issue sgl-project/sglang#30344 for the landing PRs and the up-to-date run method. We will add the command here once it ships; everything on this card runs today with the llama.cpp commands above.

Measured results (llama.cpp, RTX Pro 6000 Blackwell, greedy, 200-token completions)

Apple Silicon (M3 Max 128GB, Metal, same GGUFs, same probes)

Acceptance on Metal matches CUDA to the third decimal on byte-identical GGUFs: the drafter's behavior is backend-independent. Only the economics change per platform.

Economics across the full target-quant ladder (measured, M3 Max Metal)

SPEED-Bench (qualitative set, coding+writing, temperature 0, concurrency 1, M3 Max Metal, bf16 target)

Aggregate draft acceptance across the diverse SPEED-Bench prompts: 0.126-0.139, consistent with the greedy probes. The verdict below holds on varied real prompts, not just narrow probes.

A 0.8B target is structurally a poor speculation host: the smaller and faster the target gets, the worse speculation pays, because draft+verify overhead is roughly constant while the baseline accelerates. Break-even for this pair on Metal sits around 0.47 acceptance, which no 12K-sample head reaches. Speculation value starts around 9B-class targets (we measure 1.62x overall / 2.19x coding on a 9B with a fully-trained head, and 2.54-2.67x on a 27B: satgeze/Qwen3.6-27B-DSpark).

Loop rate: this target degenerates on long greedy generation, at every precision

Acceptance says how fast the drafter makes the target go. It says nothing about whether the target's long-form output is usable. Measured separately with an independent harness (8 prompts, greedy, 1500 tokens each, real cycle detection, prompt set v1, M3 Max Metal):

This is flat, and it is flat at the ceiling. The model loops on essentially every long greedy generation even at full bf16 precision, so here looping is a property of the 0.8B model under long greedy decoding, not a quantization effect. For contrast, the same harness on a 27B target finds zero loops at every rung down to 2.9 bpw, and only breaks below 2 bpw. Model scale, not bit-width, dominates this failure mode.

Mitigation, measured on the same rungs: enabling the DRY sampler (temperature 0.7) cuts the bf16 loop rate from 100% to 38%, but leaves Q4_K_M at 88%. So DRY helps and is worth enabling, but it does not make this target reliable for long free-form generation. Use this head for what the rest of the card describes (short completions, pipeline validation, quantization studies), not for long-form or agentic runs.

(Loop rates are only comparable within the same prompt set, token budget and decoding settings, and were all measured on Metal. Do not compare them against rates measured on another backend. Two further scoping notes: these are raw /completion probes, deliberately worst-case; chat-template serving with a repetition sampler behaves measurably better, so state which mode you measured. And if you measure a reasoning model, count reasoning tokens: a loop can live entirely inside the think field while visible content stays short, and a metric that reads only content scores a looping model as clean.)

Target quantization ladder, measured on this head

Same head (trained vs the bf16 target), same probes, only the target quantization changed. All rungs quantized from the same bf16 GGUF; the IQ1_S rung uses an imatrix computed on this model's own regeneration data (600 samples), measured on M3 Max Metal:

The bf16-trained head is effectively free down to ~4-bit targets. Below that, this 0.8B target itself stops being a coherent model and acceptance falls with it. The Q2_K row here is uncalibrated: on our 27B ladder, adding an own-distribution imatrix at the same bit-width moved acceptance from 0.149 to 0.251, and a calibrated 1.6-bpw IQ1_S 27B target held Q8-level acceptance (0.293) with the same bf16-trained-head recipe. So what collapses at the bottom of this table is the target, not the head. The same law, measured on the official deepseek-ai/dspark_qwen3_4b_block7 head in the same runtime, for reference:

Practical consequences:

1. At aggressive quantization, acceptance falls when the target itself degrades. This is a small-model effect: a 0.8B at 2.6 bpw and a 4B at 1.6 bpw are broken models, and no drafter can predict a broken model. A 27B at 1.6 bpw with a calibrated imatrix keeps Q8-level acceptance using this same bf16-trained-head recipe. Before blaming precision, check whether the target is still coherent (perplexity and loop rate both help).
2. The fix is to self-distill and train the head against the exact deployed artifact (the quantized GGUF actually served), not the full-precision base. Our llama.cpp capture tool + DeepSpec-format cache builder exist for exactly this (validated at cosine >= 0.9998 against the HF capture pipeline; full how-to guide); matched-vs-generic A/B numbers land here as they are measured. The same applies to modified variants (e.g. abliterated models): any distribution shift in the target taxes a mismatched head.
3. When evaluating published heads, compare targets at the precision the head was trained against before concluding anything about the head or the runtime.

Matched-vs-generic A/B: does capture precision matter? (measured)

We trained two additional heads with identical config/seed/steps, differing ONLY in where the training features came from: hidden states captured from the bf16 GGUF vs from the Q8_0 GGUF, both captured with our llama.cpp capture tool (this model is the testbed; caches validated at cosine >= 0.9998 against the HF pipeline). Acceptance matrix (code probe / counting probe):

Two results:

1. Pipeline validation: the bf16-GGUF-captured head reproduces the original HF-cache-trained v0.1 head (0.126/0.152 vs 0.116/0.152) within probe noise, and their final training losses are statistically identical (1.646 vs 1.63). Training from llama.cpp-captured features is a full substitute for the HF capture pipeline, proven through training, not just feature cosine.
2. No matched-head advantage at Q8: all four cells sit within probe variance. This is the expected result: Q8 features drift from bf16 by cosine ~0.999, so there is nothing for a matched head to learn. The 27B ladder later closed this out further: at 1.6 bpw with a calibrated imatrix, the same bf16-trained recipe still holds Q8-level acceptance, so the "matched training pays off at low bit-width" half of this remains a hypothesis we have not confirmed. Published matched-vs-unmatched A/Bs elsewhere measure 0-2% (arXiv 2607.04244 Table 4: 4.92 vs 4.97 accepted length; SpecForge PR #318: 7.10 vs 7.11).

Cross-stack controlled A/B: the same head in llama.cpp and vLLM (measured)

One trajectory-matched comparison, everything pinned that can be pinned: the same 8 prompts (code, math, prose), temperature 0, max_tokens 256, sequential single requests, speculative depth 7 on both stacks, bf16 target weights on both sides (HF safetensors in vLLM, the GGUF conversion of the same weights in llama.cpp), and this same step-450 head (safetensors in vLLM's --speculative-config '{"method": "dspark", ...}', the GGUF conversion via -md in llama.cpp). Acceptance computed from each stack's own raw counters (llama.cpp per-request draft acceptance log line; vLLM /metrics spec-decode counters).

Same weights, same proposals available, roughly 3.7x more drafted tokens survive verification in llama.cpp, on either denominator.

2026-07-31 correction: the 3.7x gap was a config bug in this repo, not a runtime-semantics difference. The drafter is trained with plain full-dimension RoPE, but the exported config.json inherited the target's rope geometry (partial_rotary_factor: 0.25 and mrope keys). vLLM honored those and rotated only 64 of each head's 256 rotary dims, degrading attention exactly as the vLLM row shows (position 0 partly alive, later positions near zero). llama.cpp's GGUF path never carried the bad rope keys, which is why it measured full acceptance all along. With partial_rotary_factor: 1.0 set explicitly, vLLM measures 0.155 per drafted token on the same greedy code probe vs llama.cpp's 0.158: parity. The 4.3% row is kept above as the honest record of what shipped broken, and the "runtime-semantics" framing this section carried before is retracted: the runtimes agreed once the config told the truth.

• Lesson carried forward: acceptance numbers do not transfer between stacks until the drafter's config semantics are verified against how the head was actually trained. Measure your own stack pair before committing to serving numbers.
• Throughput is deliberately not compared here. At 0.8B, speculation is a net slowdown in wall-clock terms regardless of stack (see the economics section); this A/B isolates the acceptance mechanics only.
• Reproduction note: in llama.cpp, --spec-type draft-dspark is required. Without it the server loads the draft model and then silently serves without speculation; the only visible differences are a missing draft acceptance line in the per-request timings and faster-than-expected raw decode.

Official DeepSpec eval (per-proposal acceptance, temperature 0, step 450)

Confidence head calibration is notably good even at this data scale: AUC 0.94, ECE 0.018 (mt-bench).

How this head was trained

The training signal is self-distillation: the 12K prompts (mlabonne/open-perfectblend) were answered by the target model itself (served with llama.cpp, thinking disabled, temperature 0.7), so the drafter learns the exact distribution it will draft for at inference. Hidden states were captured at layers [1, 6, 11, 16, 21] of 24 with DeepSpec's prepare_target_cache (11,577 / 11,722 samples valid), then the drafter (block size 7, 5 layers, markov_rank 256, confidence head) trained for 5 epochs / 450 steps at global batch 128 on a single 96GB GPU, loss 2.86 to 1.63. Cold start: unlike our 27B head (warm-started from a z-lab DFlash checkpoint, mean accepted length 5.4), this head was initialized from scratch, and at 12K samples that is the dominant limitation: mean accepted length ~1.8-2.05, exactly where the community scaling curve predicts a cold-start 12K head to land.

Two honest findings worth more than the head itself:

1. From-scratch heads need far more than 12K samples. The community curve (each point = that head on its own target): 2.4K samples = 1.08 accepted length (failed), 10K warm-started = 1.51, 250K = 3.34, ~1M = 4.6-4.8. v0.1 lands where the curve predicts for cold-start 12K. Warm-starting is the highest-leverage fix (our 27B head proves it at the same data scale).
2. Small targets are poor speculation hosts on fast hardware (see the economics ladder above). Small-target heads make sense on slow/edge hosts, not on datacenter GPUs.

Recipe (reproducible)

• Data: 12,000 prompts from mlabonne/open-perfectblend, responses regenerated by the target itself (llama-server bf16 GGUF, temperature 0.7, top-p 0.8, top-k 20, thinking disabled, max 2048 tokens).
• Cache: DeepSpec prepare_target_cache (capture layers [1, 6, 11, 16, 21] of 24), 11,577/11,722 valid.
• Training: DeepSpec Qwen3DSparkTrainer, block_size 7, 5 draft layers, markov_rank 256, lr 6e-4, global batch 128, 5 epochs (450 steps), bf16, single GPU. Final loss 1.63 (from 2.86).
• Conversion: llama.cpp branch satindergrewal/llama.cpp:dspark-qwen35 (adds qwen3.5-family DSpark target+drafter support on top of PR #25173), then served with --spec-type draft-dspark --spec-draft-n-max 7.

Files: head_q08_v02.gguf (v0.2 drafter, recommended, block-7, ready for llama.cpp -md), model_v02.safetensors + config_v02.json (v0.2 checkpoint, step 1020), model.safetensors (v0.1 drafter, bf16), Qwen3.5-0.8B-DSpark.gguf (converted, ready for llama.cpp -md), training config, this card.

Credits

DeepSeek (DeepSpec toolkit and DSpark method), wjinxu (llama.cpp DSpark PR #25173), ruixiang63 (DFlash/EAGLE3 foundation in llama.cpp), Ofir408 (DeepSpec online-training mode PR #23), the community heads whose published numbers anchor the scaling curve (dbirks, fal, stanleyphoong, deepseek-ai official), Qwen team (target model), mlabonne (open-perfectblend prompts).