qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont

Qwen3-4B distilled from Qwen3-30B-A3B — On-policy 5k (continual, prompts 30k–35k)

Continual on-policy knowledge distillation of a Qwen3-4B student toward the Qwen3-30B-A3B (MoE) teacher, on 5,000 math prompts from openthoughts3_math_50k (positions 30k–35k, disjoint from prior training slices).

The student for this run is RockToken/qwen3-30b-a3b-to-4b-onpolicy-10k, so the effective training exposure of this checkpoint is:

1. Off-policy KD on 20k teacher responses (positions 0–20k of openthoughts3_math_50k)
2. On-policy KD on 10k prompts (positions 20k–30k) — produced the 10k model
3. On-policy KD (this run) on 5k prompts (positions 30k–35k) — this checkpoint

Stages 2 and 3 use the same on-policy pipeline; only the seed weights, prompt slice, and step count differ.

Models

enable_thinking=False throughout (both stages).

Training data

• Source: openthoughts3_math_50k (math split of OpenThoughts-3)
• Slice used here: prompts 30k–35k → RockToken/openthoughts_prompt_math_5k_src30k-35k
• 5,000 single-user-turn prompts, mean length 261 chars, p99 845 chars

Training setup

Framework: KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.

Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).

Key hyperparameters (on-policy stage; same as stage 2)

Off-policy stage 1 hyperparameters (for context): batch 128, LR 2e-5, kd_ratio 0.5, kd_loss KL, max_len 16384, packing_samples on, ring attention.

Intended use

Research on distillation dynamics. Domain skew is math (OpenThoughts-3 math split).

Limitations

• Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
• enable_thinking=False — this student does not emit <think> traces.
• Continual training may drift capabilities relative to the base checkpoint; see the wandb curves for loss dynamics.

Weights & Biases

legendary-music-16 (stage 2, 10k) and lemon-sound-17 / 2gvd60dw (this run, stage 3) under project KDFlow, entity zhao666.