Qwen3-4B-speculator.dflash2

RedHatAI/Qwen3-4B-speculator.dflash2

This is a DFlash2 speculator model for Qwen/Qwen3-4B.

Training Details

This model was trained using the Speculators library on inference-optimization/Qwen3-8B-Regenerated-Collection. DFlash2 extends DFlash with local dynamic convolutions and a candidate selector for improved token prediction.

<details> <summary>Commands</summary>

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-4B \
  --data ./data/qwen3_8b_regen/autoif_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/evol_codealpaca_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/lmsys_arena_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/magpie_output.jsonl \
  --data ./data/qwen3_8b_regen/metamathqa_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_chat_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_math_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/nemotron_stem_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/orca_math_train_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/tulu3_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/ultrachat_output.jsonl \
  --data ./data/qwen3_8b_regen/ultrafeedback_train_sft_Qwen3-8B.jsonl \
  --data ./data/qwen3_8b_regen/ultrainteract_train_Qwen3-8B.jsonl \
  --output ./output/dflash2_qwen3_4b_8spec \
  --seq-length 8192

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0 python scripts/launch_vllm.py \
  Qwen/Qwen3-4B \
  --target-layer-ids 1 9 17 25 33 \
  -- --port 8000 \
  --max-model-len 8194 \
  --enforce-eager

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=1,2,3,4 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-4B \
  --speculator-type dflash2 \
  --data-path ./output/dflash2_qwen3_4b_8spec \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dflash2_qwen3_4b_8spec/checkpoints \
  --block-size 9 \
  --max-anchors 512 \
  --target-layer-ids 1 9 17 25 33 \
  --num-layers 5 \
  --conv-kernel-size 2 \
  --conv-group-size 16 \
  --selector-rank 256 \
  --selector-top-k 16 \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --dflash-decay-gamma 4.0 \
  --per-position-loss-weight fixed-exp-decay \
  --optimizer adamw \
  --lr 6e-4 \
  --weight-decay 0.0 \
  --scheduler-type cosine \
  --scheduler-warmup-ratio 0.04 \
  --epochs 1 \
  --total-seq-len 8192 \
  --seed 42 \
  --fsdp-shard \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.1

</details>

Model Specifications

Deployment

pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"

# Deploy with speculative decoding
vllm serve Qwen/Qwen3-4B \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-4B-speculator.dflash2",
        "num_speculative_tokens": 8,
        "method": "dflash2"
    }'

Acceptance Rates

Per-position token acceptance rates across datasets: