Qwen3.5-397B-A17B-speculator.dflash

RedHatAI/Qwen3.5-397B-A17B-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3.5-397B-A17B.

<video src="https://huggingface.co/RedHatAI/Qwen3.5-397B-A17B-speculator.dflash/resolve/main/397screenrecording.mp4" controls autoplay loop></video>

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by RedHatAI/Qwen3.5-397B-A17B and stored at Qwen3.5-397B-A17B-responses

<details> <summary>Commands</summary>

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3.5-397B-A17B
  --data ./regenerated_data.jsonl \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --output ./output \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with [vLLM](https://github.com/imargulis/vllm/tree/fix/extract-hidden-states-hybrid-block-size) installed
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3.5-397B-A17B \
  --target-layer-ids 3 15 23 35 47 \
  --max-model-len  32000 \
  --reasoning-parser qwen3 \
  --language-model-only \
  --kv-cache-dtype fp8 \ 
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 4 \ 
  --async-scheduling

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3.5-397B-A17B \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 24576 \    
  --max-anchors 3072 \    
  --target-layer-ids 3 15 23 35 47 \
  --speculator-type dflash \     
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \ 
  --draft-config 397_config.json

</details>

Model Specifications

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git  
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve Qwen/Qwen3.5-397B-A17B \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 8 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3.5-397B-A17B-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 7,                                                                                                                                                                                                                                                                   
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Latency Speedup

Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia H200:

<p align="center"> <img src="plots/speedup_HumanEval_latency.png" width="600"><br> <img src="plots/speedup_math_reasoning_latency.png" width="600"><br> <img src="plots/speedup_qa_latency.png" width="600"><br> <img src="plots/speedup_question_latency.png" width="600"><br> <img src="plots/speedup_rag_latency.png" width="600"><br> <img src="plots/speedup_summarization_latency.png" width="600"><br> <img src="plots/speedup_tool_call_latency.png" width="600"><br> <img src="plots/speedup_translation_latency.png" width="600"><br> <img src="plots/speedup_writing_latency.png" width="600"> </p>

References

Paper: DFlash: Block Diffusion for Flash Speculative Decoding