ssd-dspark-prog-qwen3-30b-a3b

dspark_prog drafter for Qwen/Qwen3-30B-A3B (ssd-parallel run)

DSpark-family drafter (dspark_prog), block size 7, 5 draft layers, target tap layers [1, 12, 23, 34, 45]. Trained on the open-perfectblend corpus regenerated by Qwen3-30B-A3B (1,339,038 cache-valid rows) with a cosine LR schedule spanning 10 epochs (max_train_steps 26150, 2615 steps per epoch). Checkpoints at epoch boundaries live under epoch_N/step_{N*2615}/; each sits at that step of the 10-epoch schedule.

Inference

Any epoch_N/step_S/config.json + model.safetensors pair loads like the DSpark paper's released drafters. Serve together with the target model Qwen/Qwen3-30B-A3B.

Resuming training

1. Rebuild the activation cache from the dataset repo xf15/ssd-perfectblend-qwen3-30b-a3b-regen with the DSpark cache builder (tokenizer/template/min_loss_tokens recorded in the cache manifest); verify the rebuild against shuffle_records/source_rows.npy — the kept-row count must equal 1,339,038.
2. Place the chosen epoch_N/step_S/ at <checkpoint_root>/deepspec/dspark_prog_block7_qwen3_30b_a3b/step_S/ and create the symlink step_latest -> step_S.
3. Resume requires world size 4 and local batch size 1 (asserted by the per-rank training_state files) and torch 2.9.1 (sampler permutations are seeded torch.randperm; shuffle_records/meta.json records the version, epoch_*.npy let you verify reproduction).

Data order is deterministic: epoch e uses torch.randperm(1339038, seed 42+e) truncated to 2615*512 samples; shuffle_records/ holds the dumped orders for all 10 epochs.