Qwen3.6-35B-A3B-DSpark-GGUF

Qwen3.6-35B-A3B DSpark GGUF

GGUF quantizations of RedHatAI DSpark draft model for Qwen3.6-35B-A3B.

Use with BeeLlama.cpp, a llama.cpp fork with advanced quantization features.

---

RedHatAI/Qwen3.6-35B-A3B-speculator.dspark

This is a DSpark speculator model for Qwen/Qwen3.6-35B-A3B.

Training Details

This model was trained using the Speculators library library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k. Responses were regenerated by Qwen3.6-35B-A3B. DSpark extends DFlash with a Markov head (intra-block token dependency) and a confidence head (per-position acceptance prediction). Training compute for this model was generously provided by Lambda, a leading cloud platform for AI training and inference.

<details> <summary>Commands</summary>

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3.6-35B-A3B \
  --data ./data/qwen3_6_35b_a3b_regen/magpie_output.jsonl \
  --data ./data/qwen3_6_35b_a3b_regen/ultrachat_output.jsonl \
  --output ./output/dspark_qwen3_6_35b_a3b \
  --seq-length 16000

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1,2,3 python scripts/launch_vllm.py \
  Qwen/Qwen3.6-35B-A3B \
  --target-layer-ids 2 10 20 30 37 \
  -- --port 8000 \
  --tensor-parallel-size 4 \
  --max-model-len 16002 \
  --enforce-eager

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 4 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3.6-35B-A3B \
  --speculator-type dspark \
  --data-path ./output/dspark_qwen3_6_35b_a3b \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dspark_qwen3_6_35b_a3b/checkpoints \
  --block-size 8 \
  --max-anchors 3072 \
  --target-layer-ids 2 10 20 30 37 \
  --num-layers 5 \
  --markov-rank 256 \
  --markov-head-type vanilla \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --epochs 5 \
  --lr 3e-4 \
  --total-seq-len 16000 \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.1

</details>

Model Specifications

Deployment

# Deploy with speculative decoding
vllm serve Qwen/Qwen3.6-35B-A3B \
    --tensor-parallel-size 2 \
    --speculative-config '{
        "model": "RedHatAI/Qwen3.6-35B-A3B-speculator.dspark",
        "num_speculative_tokens": 8,
        "method": "dspark"
    }'

Acceptance Rates

Per-position token acceptance rates across datasets: