qwen3_30b_a3b_to_4b_onpolicy_5k_src35k-40k_cont2

Qwen3-4B distilled from Qwen3-30B-A3B — On-policy 5k (continual² , prompts 35k–40k)

Second continual on-policy KD round for a Qwen3-4B student toward the Qwen3-30B-A3B (MoE) teacher, on 5,000 math prompts from openthoughts3_math_50k (positions 35k–40k).

The student for this run is RockToken/qwen3_30b_a3b_to_4b_onpolicy_5k_src30k-35k_cont, so the effective training exposure of this checkpoint is:

1. Off-policy KD on 20k teacher responses (positions 0–20k of openthoughts3_math_50k)
2. On-policy KD on 10k prompts (positions 20k–30k) → produced the 10k model
3. On-policy KD on 5k prompts (positions 30k–35k) → produced the _cont model
4. On-policy KD (this run) on 5k prompts (positions 35k–40k) → this checkpoint

Stages 2–4 share the same on-policy pipeline; only the seed weights and prompt slice differ.

Models

enable_thinking=False throughout.

Training data

• Source: openthoughts3_math_50k (math split of OpenThoughts-3)
• Slice used here: prompts 35k–40k → RockToken/openthoughts_prompt_math_5k_src35k-40k
• 5,000 single-user-turn prompts, mean length ~261 chars

Note on data overlap. The parent openthoughts3_math_50k[30k:40k] window contains 418 internal duplicate prompts across its two halves, so ~8.4% of the 5k prompts here also appeared in the prior stage-3 slice (30k–35k). This is an upstream data artifact of the OpenThoughts-3 math split, not from the slicing; the file itself is still exactly parent[5000:10000].

Training setup

Framework: KDFlow — FSDP2 + SGLang rollout, Ray-orchestrated GPU co-location with sleep/wakeup.

Hardware: 1× node, 4× H100 (80 GB), ~34 h wall-clock (~136 GPU-hours).

Key hyperparameters (identical to stages 2 and 3)

Intended use

Research on distillation dynamics and the effect of successive continual on-policy rounds on the same student. Domain: math (OpenThoughts-3 math split).

Limitations

• Trained end-to-end on math prompts only; not tuned for chat, safety, or non-math domains.
• enable_thinking=False — this student does not emit <think> traces.
• Chain-of-continual training may drift capabilities relative to earlier checkpoints in the chain; see the wandb curves for loss dynamics.