Qwen3.6-27B-DSpark-GGUF

Qwen3.6-27B DSpark GGUF

GGUF quantizations of satgeze DSpark draft model for Qwen3.6-27B.

Use with BeeLlama.cpp, a llama.cpp fork with advanced quantization features.

---

Qwen3.6-27B DSpark drafter (block-15, warm-started)

Card status (2026-07-25, corrected): all numbers below are measured. Landed: the imatrix IQ1_S and Q2_K target rungs, the matched-vs-generic A/B, the vLLM smoke test, and the official DeepSpec per-dataset eval (5 datasets). Correction: an earlier version of this card framed the target-quantization results as a "quant tax" on precision-mismatched heads. Follow-up work refuted that framing: acceptance tracks whether the target is still coherent, not the precision gap between head and target. The section below is rewritten with the mechanism and the external evidence. 2026-07-31: vLLM serving verified on this head (acceptance parity with llama.cpp), and a config.json rope fix pushed: the drafter is trained with full-dimension RoPE, but the exported config had inherited the target's partial_rotary_factor: 0.25 and mrope keys, which collapsed vLLM acceptance to ~1% for early downloaders. Fixed to partial_rotary_factor: 1.0 (explicit). Details in the vLLM section.

A DSpark speculative-decoding drafter head for Qwen3.6-27B, trained with DeepSpec in online mode against the target's own regenerated responses, warm-started from the z-lab DFlash head for this target. Served with llama.cpp (--spec-type draft-dspark) it delivers 1.6-2.7× decode speedup on an RTX Pro 6000 Blackwell at Q8_0, with zero quality change (speculative decoding is lossless: every accepted token is verified by the target).

Measured results (llama.cpp, RTX Pro 6000 Blackwell, target Qwen3.6-27B-Q8_0, greedy, 200-token completions)

SPEED-Bench (qualitative set, coding+writing, temperature 0, concurrency 1)

Aggregate draft acceptance across the diverse SPEED-Bench prompts: 0.19-0.20: the honest "varied real prompts" number, vs 0.29-0.31 on narrow greedy probes. Both are reported; when you compare drafter heads, make sure you compare the same kind of probe.

Apple Silicon (M3 Max 128GB, Metal, same GGUFs, same probes)

Acceptance is backend-independent (matches CUDA to the third decimal on byte-identical GGUFs). The economics flip with target scale: this same pipeline's 0.8B head is a net slowdown on Metal (draft overhead dominates a fast tiny target), while at 27B the head pays for itself: 1.39× on a MacBook is the difference between 12 and 17 tokens per second where it is actually felt.

How to run it

llama.cpp (tested, this is what all numbers above were measured with)

DSpark runtime support is in PR #25173 plus this model family's support on our branch. Until both are in mainline, build the branch:

git clone --branch dspark-qwen35 https://github.com/satindergrewal/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # or -DGGML_METAL=ON on Apple Silicon (default on macOS)
cmake --build build -j --target llama-server

Get the head and a target quant (Q8_0 shown; Q4_K_M measured equally good, see quant-tax below):

hf download satgeze/Qwen3.6-27B-DSpark Qwen3.6-27B-DSpark.gguf --local-dir .
hf download unsloth/Qwen3.6-27B-GGUF Qwen3.6-27B-Q8_0.gguf --local-dir .

Serve with speculative decoding:

./build/bin/llama-server \
  -m Qwen3.6-27B-Q8_0.gguf \
  -md Qwen3.6-27B-DSpark.gguf \
  --spec-type draft-dspark --spec-draft-n-max 15 \
  -c 8192 -ngl 99 -fa on --jinja --port 8080

--spec-draft-n-max 15 is not optional if you want the numbers on this card. llama.cpp defaults to drafting 3 tokens; this head was trained for a block of 15, so the default throws away most of the speedup before you ever measure it. Set it to the head's block size (15 here, printed at startup as block_size=).

Then use any OpenAI-compatible client against http://localhost:8080/v1. The server log prints draft acceptance per request. Remove -md and the two --spec-* flags to compare against baseline; output is identical (speculative decoding is lossless), only speed changes.

vLLM

vLLM has DSpark support in mainline since #46995 (merged 2026-07-01, DeepSpec-format heads) and #47093 (merged 2026-07-02, speculators-format checkpoints). Requires vLLM >= 0.25 (0.24.0 predates the merges and rejects "method": "dspark"). This repo's safetensors are DeepSpec-format with architectures: ["Qwen3DSparkModel"], exactly the key vLLM's registry maps for DSpark drafters.

Verified on this head 2026-07-31 (vLLM 0.25.1, RTX Pro 6000, bf16 target, greedy, 300-token code completion): mean accepted length 4.00, per-position acceptance 0.79 / 0.64 / 0.43 / 0.30 tapering across the 15-token block, 0.20 per drafted token. Same acceptance band as the llama.cpp numbers above. The mrope/rope code path is byte-identical between vLLM 0.25.1 and 0.26.x, so these numbers carry over.

Config note (2026-07-31 fix): this head is trained with plain full-dimension RoPE, and an earlier revision of this repo's config.json inherited the target's rope geometry (partial_rotary_factor: 0.25 plus mrope keys). vLLM honored those and rotated only a quarter of each drafter head, collapsing acceptance to ~1% (first draft position partly alive, later positions near zero). The shipped config now sets partial_rotary_factor: 1.0 explicitly; deleting the key alone is not sufficient, because transformers config classes can re-inject the 0.25 family default at parse time. If you downloaded config.json before 2026-07-31, re-download it.

vllm serve Qwen/Qwen3.6-27B \
  --speculative-config '{"method": "dspark", "model": "satgeze/Qwen3.6-27B-DSpark", "num_speculative_tokens": 15}'

Blackwell (sm_120a) setup notes from our smoke test: flashinfer device detection can fail in the engine subprocess with a misleading "requires sm75 or higher"; fix with FLASHINFER_CUDA_ARCH_LIST="12.0a". First run JIT-compiles kernels (needs ninja + nvcc on PATH). Large-vocab warmup can OOM at default settings; lower --gpu-memory-utilization / --max-num-seqs if it does.

SGLang

DSpark support is on SGLang's roadmap: watch the tracking issue sgl-project/sglang#30344 for the landing PRs and the up-to-date run method. We will add the command here once it ships; everything measured on this card runs today with the llama.cpp commands above.

Target quantization: which quants this head works for

The head was trained against the bf16 target. Measured, same head, same probes, only the target artifact changed (Metal M3 Max; acceptance is backend-identical to CUDA, verified). The imatrix rungs use an importance matrix computed on this model's own regeneration data (600 samples); in those quants the unused MTP layer (blk.64) is kept at Q8_0:

The controlled finding: at 27B scale, what decides low-bit draft acceptance is not the head, it is the quantization quality of the target. A bare Q2_K collapses acceptance (0.149); requantizing the same weights with an own-distribution imatrix recovers it (0.251-0.323), and even 1.6-bpw IQ1_S holds Q8-level acceptance (0.293) with the imatrix.

Why there is no precision tax. The DSpark paper is silent on quantization, and the architecture explains why: the drafter applies RMSNorm immediately after the fusion projection (H_ctx = RMSNorm(W_c[H^(l1); ...; H^(lm)])), which absorbs magnitude drift in the captured target features, and the drafter shares the target's frozen embedding and lm_head, so readout quantization error is common-mode between drafting and verification. Published A/Bs put matched-vs-unmatched drafter training at 0-2%: DFlash on an INT4 target scores 4.92 accepted length bf16-trained vs 4.97 INT4-trained (arXiv 2607.04244 Table 4); SpecForge measures 7.10 vs 7.11 serving FP8; poolside's own ladder rises from FP8 (5.748) through NVFP4 (5.775) to INT4 (6.273) on GSM8K, and their documented llama.cpp recipe pairs a BF16 head with a Q4_K_M target.

But acceptance alone is not a usability verdict. The loop-rate column above was measured on these exact GGUFs (8 prompts, greedy, 1500 tokens each, real cycle detection, prompt set v1, M3 Max Metal), and it falls off a cliff that neither acceptance nor perplexity shows: zero looping at

= 2.9 bpw, three quarters of prompts locked in repetition loops at 1.6 bpw. Note that an own-distribution imatrix repairs looping as well as acceptance: the same Q2_K weights go from 1/8 to 0/8. Perplexity degrades smoothly through this whole range; loop rate does not. Worse for acceptance as a metric: loops are trivially predictable, so a looping target can inflate draft acceptance. Read the three numbers together: acceptance (speedup), loop rate (long-form usability), PPL (general quality). Practical rule from this table: for long-generation or agentic workloads run >= ~2.9 bpw; below that, speculative decoding accelerates a model you would not want the output of.

Model scale dominates this failure mode. The same harness on our 0.8B target loops on 86-100% of prompts at every rung including bf16, so at that size looping is inherent to the model rather than to quantization, while this 27B stays loop-free down to 2.9 bpw. Two more measured boundaries: (1) model size matters for acceptance too, the official 4B head on an imatrix IQ1_S 4B target collapses to 0.074, so 1-bit small models are doubly hostile; (2) probe variance is large at low bpw, so treat single-probe numbers as noisy.

Sampler mitigation and depth behavior (measured)

The greedy loop column above is the worst case. Under production-style sampling (temperature 0.7 plus the DRY repetition penalty at its default multiplier 0.8), measured on the same GGUFs, same prompts, same Metal backend:

DRY keeps every rung at or above the ~2.9-bpw knee loop-free, and it does not rescue the rung below it. At 1.6 bpw the observed failures are period-1 single-token stutters (one run repeats "Ra" 200 times, another repeats a <think> tag 200 times) plus outputs collapsing to near-zero vocabulary diversity, and a sequence-level repetition penalty cannot break a distribution that has collapsed onto a single token. A sampler config is a real mitigation above the knee and no substitute for precision below it. The generation-success column exists because loop rate alone can hide the other low-bpw failure mode: output that simply dies early instead of looping.

Two scoping notes for anyone reproducing loop numbers:

• Raw-completion and chat-template numbers are not interchangeable. All rows above are raw /completion probes, deliberately worst-case. Independent measurements on a 744B production MoE with the same harness lineage found its 2.145-bpw rung loop-free at every tested depth under chat-template serving with DRY enabled, while its 1.75-bpw rung loops at shallow and deep context alike; raw-completion probes at depth can also return immediate-EOS artifacts that chat-templated serving does not show. State which mode you measured.
• Reasoning models loop invisibly. With reasoning enabled, a loop can live entirely inside the think field while the visible content stays short or empty; a loop metric that reads only content scores a looping model as clean. Count reasoning tokens too.

Where retraining against the deployed artifact is worth testing (unmeasured): targets whose weights genuinely changed rather than merely lost precision, i.e. finetunes, abliterated variants, or uncalibrated aggressive quants. The published matched-vs-unmatched deltas are 0-2%, so treat this as a hypothesis to test, not a known fix. For those, our llama.cpp capture tool + DeepSpec cache builder (validated at cosine >= 0.9998 against the HF pipeline; full how-to guide) trains the head against the exact deployed artifact. The controlled matched-vs-generic A/B is measured on the 0.8B testbed: capture-trained heads reproduce HF-pipeline training exactly, and at Q8 there is no matched-head advantage, consistent with everything above.

How llama.cpp DSpark compares with other serving stacks

Read this table honestly: rows differ in hardware, targets, and prompt sets, so it is a same-method-class comparison, not a controlled benchmark. The controlled claims are: (1) our llama.cpp implementation independently reproduces the reference acceptance for the official 4B head (0.365 vs the PR author's ~0.35), (2) accepted lengths at comparable block sizes sit inside the paper's reference band, and (3) the speedup class delivered on one consumer GPU (2.5x) and even on a laptop (1.39x) matches what vLLM's published DSpark deployment achieves on 4x B300. Speculative decoding is lossless in all stacks: output quality equals the target model's.

There is now also one properly controlled cross-stack point: the same head, same bf16 target weights, same prompts, temperature 0, spec depth 7 on both stacks, run in llama.cpp and vLLM back to back on the same GPU. llama.cpp accepted 15.8% of drafted tokens; vLLM accepted 4.3%, roughly 3.7x apart with everything material pinned. That is a runtime-semantics difference, not a head-quality difference, and it means acceptance numbers do not transfer between stacks even for the identical artifact. Full method and table: the 0.8B card.

Official DeepSpec eval (per-proposal acceptance, temperature 0, step 269)

Confidence-head calibration on these runs: AUC 0.96, ECE 0.014-0.025 per position.

Partial by choice: the run was stopped after these five datasets (the remaining four were still queued after many hours on a 27B target, and the GPU was needed elsewhere). These are the real numbers for the datasets that completed, not estimates. For comparison, the 0.8B cold-start head scores 0.10-0.29 on the same suite.

How this head was trained

The head was trained with DeepSpec's online mode (Ofir408's PR #23): instead of precomputing a multi-hundred-GB hidden-state cache, the frozen bf16 target runs its forward pass on-GPU during every training step and streams hidden states straight into the drafter. On a single 96GB card this fits the 27B target plus the drafter with micro-batch 1.

The training signal is self-distillation: the 12K prompts were answered by the target model itself (served with llama.cpp, thinking disabled), so the drafter learns to predict exactly the distribution it will be drafting for at inference time. The drafter was warm-started from z-lab's DFlash head for this target rather than initialized cold; at 12K samples this is the difference between mean accepted length ~5.4 (this head) and ~1.8 (our cold-started 0.8B head, same data scale). Three epochs, 269 optimizer steps, ~5.5 hours on one RTX Pro 6000, loss 3.27 to 1.04. Total cloud cost: $0 (local hardware).

Step-by-step, this is: regenerate responses with the served target -> train with satindergrewal/DeepSpec:qwen35-lane (config config/dspark/dspark_qwen36_27b_online.py) -> convert with satindergrewal/llama.cpp:dspark-qwen35 -> serve with --spec-type draft-dspark.

Recipe (reproducible)

• Data: 12,000 prompts (mlabonne/open-perfectblend), responses regenerated by the target itself (llama-server, Q8_0 GGUF, 16 parallel slots, thinking disabled, max 2048 tokens), 11.4K valid.
• Training: DeepSpec online mode (no precomputed cache; target forward runs per epoch on-GPU), Qwen3DSparkTrainer, block_size 15, 5 draft layers, target extract layers [1, 16, 31, 46, 61], markov_rank 256, num_anchors 128, lr 6e-4, global batch 128 (micro-batch 1), 3 epochs (269 steps), bf16, single 96GB GPU, ~5.5h. Loss 2.04 → 1.04.
• Warm-start: drafter initialized from z-lab's DFlash head for this target (shape-tolerant load), the same trick fal used: it is worth several points of acceptance at this data scale vs cold start.
• Conversion: llama.cpp branch satindergrewal/llama.cpp:dspark-qwen35 (qwen3.5/3.6-family DSpark support on top of PR #25173): convert_hf_to_gguf.py <ckpt> --target-model-dir <hf-target> --outfile head.gguf --outtype bf16
• Serve: llama-server -m Qwen3.6-27B-Q8_0.gguf -md head.gguf --spec-type draft-dspark --spec-draft-n-max 15 -fa on --jinja

Findings worth more than the head

1. Warm-starting from a published DFlash/EAGLE head is the single highest-leverage trick at small data scale. This 12K-sample head reaches mean accepted length 5.4; our identically-sized cold-start 0.8B head managed 1.8-2.05. The community scaling curve says ~250K+ samples from scratch to match this.
2. Bigger targets make speculation pay. At 27B the verify pass amortizes (mainline batch scaling is essentially linear), so acceptance 0.19-0.31 already buys 1.6-2.7×. The same head quality on a 0.8B target is a net slowdown: speculation value scales with target cost. And the head stretches the whole ladder for free, down to 1.6 bpw, as long as the quant is calibrated: IQ1_S with an own-distribution imatrix holds Q8-level acceptance. What breaks acceptance is an uncalibrated quant, not the bit-width.
3. Trainer fix shipped along the way: DeepSpec's checkpoint saver asserted micro-step alignment at end-of-data, which crashes any run whose dataset×epochs isn't divisible by the global batch (and online mode's valid-sample counts vary per epoch, so this is unavoidable). Patched to align the final save down to the last completed optimizer step: satindergrewal/DeepSpec:qwen35-lane.
4. Sampler mitigation stops at the precision knee. Temperature 0.7 with DRY keeps every rung from Q8 down to 2.9-bpw Q2_K+imatrix loop-free on this target, and still loses 5 of 7 prompts on 1.6-bpw IQ1_S to period-1 single-token loops a sequence penalty cannot touch. Above the knee, looping is a sampling artifact; below it, distribution collapse.

Files

model.safetensors (drafter, bf16), Qwen3.6-27B-DSpark.gguf (block-15 head, ready for llama.cpp -md), training config, this card.

Credits

DeepSeek (DeepSpec toolkit and the DSpark method), z-lab (DFlash warm-start weights), wjinxu (llama.cpp DSpark PR #25173), ruixiang63 (DFlash/EAGLE3 foundation in llama.cpp), Ofir408 (DeepSpec online-training mode PR #23), hikarioyama (warm-start recipe), Qwen team (target model), mlabonne (open-perfectblend prompts).

Footnotes

1. provenance note: this Q8_0 row was measured on a community-published GGUF, while every other rung was quantized here from one bf16 conversion. Those two chains differ in quantization recipe and, notably, the community Q8_0 omits the model's MTP block entirely (65 blocks vs 64). That does not affect DSpark drafting, which reads trunk layers only, but it means the Q8_0 row is not from the same source chain as its neighbours. ↩