gemma-4-31B-it-speculator.dflash.swa.causal.fullvocab.muon

Training

Prepare the data:

python scripts/prepare_data.py \
  --model google/gemma-4-31B-it \
  --data magpie_gemma-4-31B-it.jsonl \
  --data ultrachat_gemma-4-31B-it.jsonl \
  --output ./data \
  --num-preprocessing-workers 32

Launch the hidden-state server (GPUs 0-3):

CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  google/gemma-4-31B-it \
  --hidden-states-path ./hidden_states \
  --target-layer-ids 1 17 29 47 58 \
  -- \
  --port 8887 \
  --max-model-len 16384 \
  -tp 2 \
  --data-parallel-size 2 \
  --no-enable-chunked-prefill

Train (GPUs 4-7):

CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun --standalone --nproc_per_node=4 scripts/train.py \
  --speculator-type dflash \
  --verifier-name-or-path google/gemma-4-31B-it \
  --data-path ./data \
  --hidden-states-path ./hidden_states \
  --on-missing generate --on-generate delete \
  --vllm-endpoint http://localhost:8887/v1 \
  --target-layer-ids 1 17 29 47 58 \
  --num-layers 5 \
  --max-anchors 512 \
  --block-size 8 \
  --sliding-window 2048 \
  --scheduler-type cosine \
  --epochs 3 \
  --noise-std 0.0 \
  --optimizer muon \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --fsdp-shard \
  --logger trackio \
  --run-name gemma-4-31B-it-dflash-swa-causal-fullvocab-muon \
  --save-path ./checkpoints

Deployment

Serve the model:

CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.openai.api_server \
  --model google/gemma-4-31B-it \
  --speculative-config '{"model": "inference-optimization/gemma-4-31B-it-speculator.dflash.swa.causal.fullvocab.muon", "method": "dflash", "num_speculative_tokens": 7}' \
  -tp 2 \
  --max-model-len 16384 \
  --port 8001

Run benchmarks:

cd speculators/scripts/evaluate && python evaluate.py \
  --target http://localhost:8001/v1 \
  --output-dir ./epoch3_throughput \
  throughput

Evaluation results