Qwen2.5-7B-Instruct-DirectOPD-R1DistillShift-100

Qwen2.5-7B-Instruct — Direct-OPD transfer of the R1-distillation shift, 100 steps

Artifact of the Direct-OPD SFT-transfer experiment (direct-opd-sft-transfer, condition r1distill). The question: when a pure-SFT shift encodes real held-out capability, does Direct-OPD transfer that capability — and into a student 4.7x larger than the teachers?

The training signal is the token-level shift log pi_post - log pi_pre, evaluated on the student's own sampled tokens. Neither teacher is imitated; only the difference between them is.

Root = step 100. checkpoint-{20,40,60,80,100}/ = intermediate merged checkpoints. Weights are bf16 (verl's FSDP->HF merge downcasts the fp32 masters).

Training configuration

• code: https://github.com/BytedTsinghua-SIA/Direct-OPD @ 3a9d6bd37b00a38e7a9b2959239e4631e5324aea + logs/phase4_seed.patch (seed 42 shim)
• data: cmpatino/direct-opd-sft-deepmath-pilot-data @ 22625ae5db434947195bf862c429cd94504a4809 :: opd_train.parquet (6,400 AIME-decontaminated prompts, one pass)
• 100 steps x 64 prompts x 4 rollouts, lr 1e-6, adaptive KL (init/max 2.5, min 0.5), token-mean loss, fp32 masters
• sequence budget 768 prompt + 3328 response = 4096, pinned to pi_pre's max_position_embeddings (4096): the reward must never score a position the pre-teacher was never trained on
• top-k 16 only_stu, student_p weighting, T=1.0 for student and teachers, reward_model.model.input_tokenizer=null (teachers score the student's rendered ids verbatim)
• 4 x A100-80GB, FSDP full-shard, gradient checkpointing + activation offload
• driver + full env block: logs/run_manifest.json, console log logs/train.log.gz

Caveats

• No in-training validation (test_freq=-1); all evaluation is external and pre-registered.
• Bit-exact reproducibility is not attainable (vLLM continuous batching, dynamic micro-batching, FSDP reduction order). Seeding fixes data order and engine seeds only.
• Prompts are rendered with the student's chat template in training and in evaluation (verl RLHFDataset, rl_dataset.py:363), so Qwen2.5's default system prompt is present in both — consistent by construction.
• Special ids 151643-151649 carry different meanings in the DeepSeek post-teacher than in the Qwen pre-teacher/student; only the response's terminal <|im_end|> is affected (~1 token/response). Reported, not fixed.