Qwen3.8-27B-Dspark-v1

Qwen3.8-27B-DSpark

⚡ Day-1 support for Qwen3.8-27B

Trained, benchmarked and released within a day of the base model going public -- speculative decoding you can turn on immediately, not months later.

A DSpark speculative-decoding drafter head for Qwen3.8-27B, trained against the Q4_K_M GGUF that actually gets served rather than the bf16 checkpoint. On a single RTX 4090D it delivers 1.69x-2.51x decode speedup depending on the workload.

Measured decode speed (llama.cpp, 1x RTX 4090D 48G, Q4_K_M target, temperature 0)

All throughput numbers below are decode tokens/s (llama.cpp predicted_per_second), measured on generation only -- prompt processing is excluded. Batch size 1, 256 new tokens.

Baseline is the same target with no draft: 48.86 decode tok/s (stable within +/-0.06 across all nine sets, so the differences below are the workload, not measurement noise).

Structured reasoning (math) speculates best; open-ended writing worst. Competitive programming (livecodebench) is notably harder than textbook code (humaneval), which is worth knowing if your workload looks more like the former.

Works across quantizations

The head was trained on hidden states captured from Q4_K_M, but it transfers to other quantizations of the same target without retraining. Same draft, same prompts, only the target file changed:

Two things worth knowing:

Accepted length barely moves (3.230 vs 3.266, well under 1%) even across a 2x precision change. The drafter conditions on the target residual stream at five layers, and those intermediate representations stay consistent across quantizations, so a head trained on one quant works on the others.

Higher precision gets a larger relative speedup (2.14x -> 2.37x) because the non-speculative baseline slows down with weight size (46.18 -> 31.36 t/s) while speculation amortizes weight reads across several verified tokens, trading a memory-bandwidth bottleneck for a compute one. Absolute throughput is still higher on the 4-bit file (98.84 vs 74.44 t/s), so pick UD-Q4_K_XL for speed and Q8_0 for quality.

Per-dataset speedup range: UD-Q4_K_XL 1.80x-2.64x, Q8_0 2.00x-2.99x.

Usage

Requirements

Upstream ggml-org/llama.cpp -- pull the latest master and build. No fork needed.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-server

Running

llama-server \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  -md Qwen3.8-27B-DSpark-Q8_0.gguf \
  --spec-type draft-dspark \
  --spec-draft-n-max 4 \
  -ngl 99 -ngld 99 -c 8192

-ngl 99 offloads the target, -ngld 99 the draft -- both are needed; leaving the draft on CPU erases the speedup.

--spec-draft-n-max is the one flag you must set. The default of 3 is close, but the optimum depends on the head, not on the target:

Accepted length keeps rising past n_max=4, but the per-step drafting cost rises faster, so throughput peaks at 4. If you swap in a different head, re-sweep -- a head with longer accepted length will peak at a higher n_max.

Use the Q8_0 file: same accepted length as bf16, about 7% faster, half the size.

Files

model.safetensors deliberately excludes embed_tokens and lm_head: those are frozen copies of the target weights (2.54B params, 58% of the raw checkpoint). convert_hf_to_gguf.py --target-model-dir <target> pulls them from the target, so nothing is lost and the target weights are not redistributed.

Training

Warm-starting from the 3.6 head rather than a DFlash head works because Qwen3.6 and Qwen3.8 share the same config (except transformers_version) and the same vocab.json/merges.txt, so the weights transfer directly.

Future work

This head was trained on 40,000 samples, which is where the current results come from rather than any ceiling of the method. Data volume is the lever we have seen move the number most: on Qwen3.6-27B, going from ~11K to 35K target-regenerated samples raised mean accepted length from 2.27 to 3.46 on the same evaluation set. We plan to:

• Train on substantially more data. Accepted length was still improving with scale at 40K, so the current numbers are a floor, not a limit.
• Release the training pipeline. The end-to-end path -- answer regeneration from the served quantized target, hidden-state capture from the GGUF, sharded multi-GPU capture, and the consumer-GPU patches needed to train these heads on 4090-class cards -- will be published so others can build heads for their own targets and quantizations.
• Cover more targets and quantizations. Same recipe, different base models.

Follow this repo for updates.