Qwen3-8B-speculator.dflash2

RedHatAI/Qwen3-8B-speculator.dflash2

This is a DFlash2 speculator model for Qwen/Qwen3-8B, trained without reasoning (enable_thinking=False).

Training Details

This model was trained using the Speculators library on a mix of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered, the train_sft split of HuggingFaceH4/ultrachat_200k, and nvidia/Nemotron-Personas data. Responses were regenerated by Qwen3-8B (without reasoning). Training uses D-PACE loss weighting and the Muon optimizer, with a re-warm cosine learning rate schedule for continued improvement after initial convergence.

<details> <summary>Commands</summary>

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

python scripts/prepare_data.py \
  --model Qwen/Qwen3-8B \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 16384

Launch vLLM

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-8B \
  --target-layer-ids 3 11 19 27 35 43 \
  -- --port 8000 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --disable-uvicorn-access-log

Launch training (resumed with re-warm cosine LR)

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-8B \
  --speculator-type dflash2 \
  --block-size 16 \
  --num-layers 8 \
  --data-path ./output \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/checkpoints \
  --epochs 7 \
  --lr 6e-4 \
  --total-seq-len 16384 \
  --max-anchors 768 \
  --target-layer-ids 3 11 19 27 35 43 \
  --on-missing generate \
  --on-generate delete \
  --loss-fn ce \
  --per-position-loss-weight dpace \
  --dpace-alpha 0.5 \
  --optimizer muon \
  --muon-lr 0.003 \
  --muon-momentum 0.95 \
  --muon-weight-decay 0.1 \
  --muon-ns-steps 5 \
  --weight-decay 0.01 \
  --scheduler-type cosine \
  --scheduler-total-steps 68922 \
  --prefetch-factor 2 \
  --num-workers 8 \
  --log-freq 100

</details>

Model Specifications

Deployment

pip install vllm

vllm serve Qwen/Qwen3-8B \
    --tensor-parallel-size 1 \
    --max-model-len 16384 \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-8B-speculator.dflash2",
        "num_speculative_tokens": 7,
        "method": "dflash2"
    }'

Evaluations

Per-position token acceptance rates on the RedHatAI speculative decoding benchmark (native 7 draft tokens, enable_thinking=False, max_tokens=512, temperature=0):

References

Paper: DFlash2: Block Diffusion for Improved Speculative Decoding