qwen3-8b-domino-draft-step44000

Qwen3-8B Domino speculative-decoding draft (10-epoch)

A Domino draft model for Qwen/Qwen3-8B, trained with the SpecForge DataFlow online-disaggregated pipeline (frozen Qwen3-8B target + live spec-capture through sglang/Mooncake -> DP domino trainer).

This checkpoint

• training_state.pt — torch checkpoint; the trained draft weights are under the key draft_state_dict (~1.1B params, bf16), plus training metadata (global_step, epoch, strategy, run_id, ...).
• config.json — the draft config (DFlashDraftModel, projector_type=domino, 5 layers, hidden 4096, target_layer_ids=[1,9,17,25,33], block_size=16, vocab 151936, mask_token_id=151669).
• Step 44,000 — the latest periodic checkpoint (save-interval 4000) of a completed 10-epoch run (~epoch 9.3).

Training

• Data: perfectblend (47,160 filtered prompts), 10 epochs, batch 2, DP=5, LR 6e-4, warmup 0.04, grad-clip 1.0, num_anchors 256, loss-decay-gamma 7.0.
• Target Qwen/Qwen3-8B frozen (no gradients); only the draft is trained.
• Final training-time metrics: loss ~0.1, token-match acc ~0.95.

Note: the training-time acc is a proxy. Real serving accept-length (τ) requires exporting to an sglang-loadable draft and running the spec-decode benchmark.

Load the draft weights

import torch
sd = torch.load("training_state.pt", map_location="cpu", weights_only=False)
draft_weights = sd["draft_state_dict"]  # load into a DFlashDraftModel(config)