qwen3-4b-dpard-thinking-b16

qwen3-4b-dpard-thinking-b16

DFlash draft model (3 layers, block size 16) for speculative decoding of Qwen/Qwen3-4B in thinking mode.

Training objective: D-PARD: full-vocabulary Rényi-half divergence with detached position weights from exact rejection-sampling acceptance (dpard_alpha = 0.5).

Training data

jgeuter/ShareGPT-Qwen3-4B-T0.6-Thinking-Regen: 36,315 ShareGPT conversations regenerated by Qwen3-4B with thinking enabled (T=0.6, top-p 0.95, top-k 20, 32k token budget), expanded to 101,212 per-turn training samples (only the final assistant turn of each sample is supervised, reasoning included; chat template with thinking enabled; max sequence length 8192).

Training setup

SpecForge (offline feature capture from Qwen3-4B layers [1, 17, 33]), AdamW, lr 6e-4 cosine with 4% warmup, global batch 4, 6 epochs, 512 anchors per sequence, grad clip 1.0, bf16, seed 42. Matches the D-PARD/D-PACE paper recipe except sequence length (8192 vs 3072) and the thinking-mode corpus.

Usage

Serve with SGLang:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-4B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path jgeuter/qwen3-4b-dpard-thinking-b16 \
  --reasoning-parser qwen3

These are research artifacts for comparing DFlash / D-PACE / D-PARD training objectives on thinking-mode data.