deepspec-drafter-ablations

DeepSpec drafter ablations — budget-matched one-epoch arms

Training checkpoints for the ablation arms behind TIE-Pilot/dspark-attnconv-block7-qwen3-4b, the drafter reported in arXiv:2608.27339.

Every arm here ran exactly one epoch — 2,616 optimizer steps — at lr 6e-4 with 512 anchors on the same data, against the same frozen Qwen/Qwen3-4B target. The arms differ only along the axis each is named for, so the budget is matched by construction and the comparison is not confounded by training length.

What each arm changes

Measured at this budget

Accepted length on 430 temperature-zero rows, per-task cap 50, maximum generation length 2048:

The reproduced DFlash2 is below vanilla at this budget. The head and the short convolution both help, and the combined variant is best. Slot embeddings alone are essentially unchanged; they are kept because they cost nothing at inference and appeared to speed up convergence.

These rows do not isolate final component contributions. The head arm changes the head architecture and adds the nomination objective, and its initial output scale is ~0.0426 while the combined arm uses 0.35. Read them as design comparisons, not as an additive decomposition.

One epoch is not enough to judge an architecture. The same recipe reaches 87% of its converged accepted length after a single epoch, so a change that only pays off late looks like noise here. The per-epoch trajectory in the flagship repository shows how much moves after this point.

The anchor column

The head and combined arms both carry a separate anchor key/value column (markov_head.anchor_type, markov_head.anchor_norm.weight). Removing it at evaluation does not affect acceptance, and keeping it made longer runs collapse — the first attempt at the 10-epoch run diverged with that column's gradient scale 13.7x a healthy control's after median normalisation, against a 2.45x ceiling across the other 118 tensors. The released flagship sets markov_anchor_kv=False. If you train from these checkpoints for more than an epoch, turn it off.

Layout and loading

<arm>/step_2616/config.json, model.safetensors, train_config.py, training_metadata.json

train_config.py is the exact configuration the trainer wrote for that arm — diff two of them to see precisely what an axis changed.

Optimizer state is not included: every arm here ran its cosine schedule to the end and its scheduler records _last_lr == 0, so resuming as-is would take zero-sized steps. Continuing any of these arms means defining a new schedule regardless.

Qwen3DSparkModel is a custom block-parallel drafter, not a transformers architecture, so AutoModel.from_pretrained will not resolve it; loading needs the DeepSpec code. The safetensors files are otherwise standard.

Citation

This work is described in Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting (arXiv:2608.27339). The paper separates a block drafter's rejection into an information floor — the minimum expected rejection at a given conditioning order — and the model gap above it, which is the part a better proposal can actually recover. The prefix-attention head in this repository is the design that distinction points to: it reads committed context conditional on the predecessor, rather than applying a context-independent predecessor correction.

@misc{qiang2026parallelblindnessinformationfloors,
      title={Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting}, 
      author={Xinwei Qiang and Xiang Fang and Chang Chen and Zaifeng Pan and Yue Guan and Yufei Ding},
      year={2026},
      eprint={2608.27339},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.27339}, 
}