gemma-4-31B-it-DSpark-GGUF

gemma-4-31B-it DSpark GGUF

GGUF quantizations of RedHatAI DSpark draft model for gemma-4-31B-it.

Use with BeeLlama.cpp, a llama.cpp fork with advanced quantization features.

---

RedHatAI/gemma-4-31B-it-speculator.dspark

This is a DSpark speculator model for google/gemma-4-31B-it.

Training Details

This model was warm-started from a DFlash checkpoint and trained using the Speculators library. Training compute for this model was generously provided by Lambda, a leading cloud platform for AI training and inference.

<details> <summary>Commands</summary>

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model google/gemma-4-31B-it \
  --data ./regenerated_data.jsonl \
  --output ./output \
  --seq-length 16384

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
  google/gemma-4-31B-it \
  --target-layer-ids 1 17 29 47 58 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --tensor-parallel-size 2 \
  --no-enable-chunked-prefill

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=2,3 torchrun \
  --standalone \
  --nproc_per_node 2 \
  scripts/train.py \
  --verifier-name-or-path google/gemma-4-31B-it \
  --speculator-type dspark \
  --from-pretrained <dflash-checkpoint> \
  --data-path ./output \
  --on-missing generate \
  --on-generate delete \
  --scheduler-type cosine \
  --draft-vocab-size 32000 \
  --max-anchors 1024 \
  --target-layer-ids 1 17 29 47 58 \
  --num-layers 5 \
  --lr 1e-4 \
  --backbone-lr-scale 0.1 \
  --epochs 3 \
  --sliding-window 2048 \
  --markov-rank 256 \
  --markov-head-type vanilla \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --draft-hidden-act silu

</details>

Model Specifications

Deployment

# Deploy with speculative decoding
vllm serve google/gemma-4-31B-it -tp 4 --spec-model RedHatAI/gemma-4-31B-it-speculator.dspark --spec-tokens 7 --spec-method dspark

Acceptance Rates

Per-position token acceptance rates across datasets:

Performance Eval