Qwen2.5-7B-Instruct-DirectOPD-RAFTShift-100

Qwen2.5-7B-Instruct — Direct-OPD transfer of the OpenThinker3 reasoning-SFT shift, 100 steps

Artifact of the Direct-OPD SFT-transfer experiment (direct-opd-sft-transfer), extension condition openthinker. H2 — with a chat→reasoning SFT pair the student gains on at least one held-out benchmark (paired CI excluding 0).

The training signal is the token-level shift log pi_post - log pi_pre, evaluated on the student's own sampled tokens. Neither teacher is imitated; only the difference between them is.

Because pi_post was fine-tuned from pi_pre, this shift isolates reasoning SFT alone — unlike the experiment's first run, whose base→SFT pair also bundled "became a chat model".

Root = step 100. checkpoint-{20,40,60,80,100}/ = intermediate merged checkpoints. Weights are bf16 (verl's FSDP->HF merge downcasts the fp32 masters).

Training configuration

• code: https://github.com/BytedTsinghua-SIA/Direct-OPD @ 3a9d6bd37b00a38e7a9b2959239e4631e5324aea + logs/phase4_seed.patch (seed 42 shim)
• data: cmpatino/direct-opd-sft-deepmath-pilot-data @ 22625ae5db434947195bf862c429cd94504a4809 :: opd_train.parquet (6,400 AIME-decontaminated prompts, one pass)
• 100 steps x 64 prompts x 4 rollouts, lr 1e-6, token-mean loss, fp32 masters
• KL coefficient: adaptive, init 2.5, clamped to [0.5, 2.5], eps 0.01 (x(1−eps) after a negative mean shift reward, x(1+eps) after a positive one)
• sequence budget 768 prompt + 3328 response = 4096 (both teachers allow 32,768 positions, so this is a deliberate rollout budget, not a context limit)
• top-k 16 only_stu, student_p weighting, T=1.0 for student and teachers, reward_model.model.input_tokenizer=null (teachers score the student's rendered ids verbatim)
• 4 x A100-80GB, FSDP full-shard, gradient checkpointing + activation offload, actor optimizer offload True, log-prob token budgets ref/rollout 8192/8192
• driver + full env block: logs/run_manifest.json, console log logs/train.log.gz

Caveats

• No in-training validation (test_freq=-1); all evaluation is external and pre-registered.
• Bit-exact reproducibility is not attainable (vLLM continuous batching, dynamic micro-batching, FSDP reduction order). Seeding fixes data order and engine seeds only.
• Prompts are rendered with the student's chat template in training and in evaluation (verl RLHFDataset, rl_dataset.py:363).
• The trio's tokenizers agree over the whole defined id range (P0 gate: ordinary ids [0, 151642] identical, added tokens identical, chat templates byte-identical), so the cross-tokenizer caveat that applied to this experiment's first run does not apply here.
• Qwen2.5-7B-Instruct's template injects its default system prompt in training and in evaluation (same apply_chat_template(..., add_generation_prompt=True) call) — consistent by construction.