Qwen3-4B-DSpark-b16

Qwen3-4B-DSpark-b16

A DSpark speculative-decoding draft model for Qwen/Qwen3-4B, fine-tuned to a block size of 16 (the released DSpark checkpoints for this target are block 7).

This is a drafter, not a standalone language model. It proposes tokens that a Qwen3-4B verifier accepts or rejects; used correctly, output is bit-identical to plain autoregressive decoding from the target.

Why block 16

DSpark runs its backbone once in parallel over a block of block_size positions, then applies a small rank-256 markov head in a short serial sweep so draft token k+1 is conditioned on draft token k. A longer block means a longer draft horizon per verifier call, but per-position acceptance decays along the block. The markov head is what slows that decay, so DSpark is exactly the architecture where pushing the block out to 16 should still pay — and it is the regime that benefits most from tree-structured verification, which needs depth to have anything to branch over.

Training

Warm-started from deepseek-ai/dspark_qwen3_4b_block7 and fine-tuned with DeepSpec (MIT).

Warm-starting across block sizes is valid because the drafter's parameter shapes are block-size-agnostic — block_size only controls how many mask tokens are appended per anchor during training — so the block-7 weights load under load_state_dict(strict=True) with no surgery.

embed_tokens and lm_head are copied from the target and frozen; only the backbone, markov head, and confidence head are trained.

Why this is the smaller-data checkpoint

A larger training arm (9,500 conversations vs 2,280) scored worse on all 6 evaluation datasets, by −0.5% to −6.7%, so the smaller run is what ships.

That is a selection criterion, not a finding about data scale. The larger arm changed five things at once — conversations 2,280→9,500, sequence length 768→512, anchors 32→96, steps 600→1400, loss decay γ 4→8 — so the regression is unattributable, and the shortened context is at least as plausible a cause as the extra data. Whether data is the binding constraint here is untested.

Separately, the run with the lowest training loss (0.522) was not the best drafter: 1000 steps over 2,280 conversations is 31 epochs, and it lost on acceptance. Train loss is not the metric for a drafter; accepted tokens per verifier call is.

Results

From the prior evaluation phase — H100, greedy (temperature 0), exact-match prefix acceptance, 12 prompts x 512 tokens x 6 datasets (humaneval, mbpp, gsm8k, math500, mt-bench, alpaca). acc = accepted tokens per verifier call, mean across datasets:

Read that as the end-to-end gain of the block-16 drafter plus tree verification over the released block-7 drafter decoding natively — it does not isolate block size from the verification strategy. A controlled block-7-vs-block-16 comparison holding the tree builder fixed is the subject of experiment 2 in the repo below; those numbers are not in yet, and this card will be updated when they are rather than being back-filled from the table above.

Repeated runs of an identical configuration in that harness differ by ~1% on acceptance and ~5% on wall-clock; deltas inside those bands are not effects. All numbers are batch 1 — acceptance is the axis that transfers to batched serving, wall-clock at batch 1 is not.

Usage

The model loads through the DSparkDraftModel port in the repo below. Two things to know:

• It is not loadable with plain AutoModelForCausalLM — Qwen3DSparkModel is not a transformers architecture. Use the harness.
• The config specifies RoPE in the transformers-v5 style (rope_parameters) with no top-level rope_theta. Under transformers 4.x, config.rope_theta silently falls back to the 10000.0 default instead of the 1000000.0 this model was trained with, which degrades drafts without raising. Either use transformers

  = 5, or normalize the field before instantiating the model.

from transformers import Qwen3Config
from model import DSparkDraftModel   # from the repo below

config = Qwen3Config.from_pretrained("shreybirmiwal/Qwen3-4B-DSpark-b16")
# transformers 4.x compatibility, see note above
if getattr(config, "rope_parameters", None):
    config.rope_theta = config.rope_parameters["rope_theta"]

draft = DSparkDraftModel.from_pretrained(
    "shreybirmiwal/Qwen3-4B-DSpark-b16",
    config=config,
    attn_implementation="flash_attention_2",
    dtype="bfloat16",
).cuda().eval()

Links

• Code, training recipe, and benchmarks: https://github.com/jwlaboratory/diffusion-sparked-tree
• Training framework: DeepSpec
• Base checkpoint: deepseek-ai/dspark_qwen3_4b_block7