dspark-attnconv-block7-qwen3-4b

DSpark-AttnConv block-7 drafter for Qwen3-4B

A 5-layer block-parallel speculative-decoding drafter for Qwen/Qwen3-4B. One drafter forward proposes a block of 7 tokens; the target model verifies them in a single pass.

This is the drafter reported in arXiv:2608.27339 — the +3.83% mean serving accepted length over the released DSpark checkpoint quoted in that abstract is the temperature-1 macro figure in the table below.

Relative to a DSpark-style baseline of identical size and training budget, this drafter adds three things: an attention-style order-1 head (the predecessor-conditioned head is a real attention block rather than a low-rank bilinear form), a depthwise short convolution over the draft-horizon axis, and slot embeddings on the residual stream.

Results

Measured against the official deepseek-ai/dspark_qwen3_4b_block7 drafter on the same benchmark rows, same harness, same target model. τ is mean accepted length per verify step; S is end-to-end speedup over autoregressive decoding.

Higher on all nine tasks, at both temperatures, on both metrics. Temperature-1 acceptance averages three independent launches because sampling is stochastic: 4.9329 ± 0.0139 for ours against 4.7511 ± 0.0092 for DSpark, a relative difference of 3.83% ± 0.21 percentage points (one propagated standard error across launches).

Losslessness requires --enable-deterministic-inference. Speculative decoding is token-exact only under deterministic reduction order. Without that flag, acceptance patterns change which tokens are batched into each verify pass, that changes the target's reduction order, and ties flip — so output can differ from autoregressive decoding even though the accept/reject rule itself is exact.

Training trajectory

The run is a single 10-epoch schedule, and this repository ships one checkpoint per epoch. Accepted length at temperature 1, measured before serving optimizations:

Gains flatten late: epochs 7, 9 and 10 give 4.9221, 4.9367 and 4.9395. One epoch alone already reaches 87% of the converged value, which is worth knowing before attributing a small delta to an architectural change — a 1-epoch screen cannot see a change that only pays off late.

Repository layout

config.json, model.safetensors     epoch-10 weights, for loading directly
step_2616/                         epoch 1   ┐ each: config.json,
step_5232/                         epoch 2   │ model.safetensors,
...                                          │ train_config.py,
step_26160/                        epoch 10  ┘ training_state.rank0.pt

Every per-epoch directory carries its optimizer state, so training can be resumed from any epoch boundary, and train_config.py records the exact configuration that produced it.

Architecture

The embedding and LM head are the target's, frozen throughout; they are included in the checkpoint so it loads standalone.

Training

The objective is TV-weighted rather than pure cross-entropy on purpose. Cross-entropy to the target's argmax is a temperature-0 objective; the 0.1/0.9 mix tracks the acceptance probability itself and wins on both greedy and temperature-1 rulers.

Loading

Qwen3DSparkModel is not in transformers — this is a custom block-parallel drafter architecture, and AutoModel.from_pretrained will not resolve it. Loading it requires the DeepSpec training/serving code, and running it end-to-end requires an SGLang build with the matching block-parallel speculative path. The safetensors file and config.json are otherwise standard, so the weights can be inspected with any safetensors reader.

Note that --speculative-num-draft-tokens means different things in different speculative backends: in the DFLASH path the flag is the block size (γ = block_size − 1, position 0 being the anchor), not the number of speculative slots. Passing the wrong convention silently grants an extra slot and inflates accepted length.

Citation

This work is described in Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting (arXiv:2608.27339). The paper separates a block drafter's rejection into an information floor — the minimum expected rejection at a given conditioning order — and the model gap above it, which is the part a better proposal can actually recover. The prefix-attention head in this repository is the design that distinction points to: it reads committed context conditional on the predecessor, rather than applying a context-independent predecessor correction.

@misc{qiang2026parallelblindnessinformationfloors,
      title={Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting}, 
      author={Xinwei Qiang and Xiang Fang and Chang Chen and Zaifeng Pan and Yue Guan and Yufei Ding},
      year={2026},
      eprint={2608.27339},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.27339}, 
}