GLM-5.2-speculator.dspark

RedHatAI/GLM-5.2-speculator.dspark

This is a preliminary (and subject to change) DSpark speculator model for RedHatAI/GLM-5.2-NVFP4-FP8.

It was trained using the Speculators library on the mgoin/open-perfectblend-glm5.2-regen dataset.

Note:

It was validated on Nvidia B200, other hardware validation pending.

Training Details

Launch training

scripts/train.py \
  --verifier-name-or-path RedHatAI/GLM-5.2-NVFP4-FP8 \
  --data-path ./output/dspark_glm52 \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dspark_glm52/checkpoints \
  --epochs 1 \
  --lr 3e-4 \
  --total-seq-len 8192 \
  --speculator-type dspark \
  --block-size 8 \
  --max-anchors 1024 \
  --num-layers 3 \
  --target-layer-ids 2 20 39 58 75 \
  --markov-rank 256 \
  --markov-head-type vanilla \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --fsdp-shard \
  --logger tensorboard \
  --log-dir ./output/dspark_glm52/logs \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.01 \
  --request-timeout 60 \
  --max-retries 2

Deployment

Deploy with vLLM using the speculator as a draft model.

vllm serve RedHatAI/GLM-5.2-NVFP4-FP8  \
  --max-model-len 20480 \
  --spec-model RedHatAI/GLM-5.2-speculator.dspark \
  --spec-method dspark \
  --spec-tokens 7 \
  -tp 8

Preliminary Evaluations

Per-Position Acceptance Rate