Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176

Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176

Qwen/Qwen3-4B-Instruct-2507 fine-tuned with RL to write short "thoughts" about the text that immediately follows a cut in a stat.ML arXiv paper. This is the step 2176 policy of the prestar-RL run reason-only / stride-1 consumer / lr 7e-7, no KL, exported in fp32 from the trainer's FSDP master weights. It is the best-validation checkpoint of the run at the time of export.

What it was trained to do

Given a document prefix, the policy writes a few dense sentences inside <thought> … </thought> about the next 8 tokens. The reward is the mean per-token log-probability of the true 8-token continuation under a frozen suffix conditional, JackHsieh/4B-base.4B-Instruct-reason-thoughts.stride-1.suffix-conditional, which reads the thought in a KEY/VALUE hint wrap prepended to the document. Higher is better; the base model scores -1.1493 nats/token on the validation grid and this checkpoint scores -1.1420.

Training

• Algorithm: RLOO (leave-one-out baseline), 8 samples per prompt, 64 prompts per step, CISPO-style policy loss, no KL term
• Optimizer: AdamW, constant lr 7e-7
• Prompts: the first 1,024 documents of JackHsieh/statML-arxiv-RL-4k-docs, every 8-token chunk (stride 1)
• Sampling during training: temperature 0.6, top-p 0.9, stop at </thought>, 1,024-token cap; a rollout with no </thought> receives a fixed penalty of -8 nats/token
• Validation: one sample per chunk on the stride-32 grid of the JackHsieh/statML-arxiv-40M-20M test split (72,960 chunks)
• Framework: verl with vLLM rollouts; 8x H200
• Logs: wandb run (the run was requeued several times; each segment is a separate wandb run in the same group)

Usage

Prompt with the reason-only chat template used for training: a system message describing the task, the raw document prefix as the user turn, and an assistant turn pre-filled with <thought>\n. Sample at temperature 0.6, top-p 0.9, and stop on </thought>. The template is in the prestar repository (chat_templates/qwen/reason-only-nothink.jinja).

<!-- Internal provenance (not needed to use the model): run config prestar/configs/rl/experiment/prestar_rl/qwen-4B/reason-only-stride1-consumer.yaml overrides lr=7e-7 kl_loss_coef=0.0 total_training_steps=14080 test_freq=64 save_freq=32 gpu_memory_utilization=0.7 slurm job 17454160 (miso); wandb group rl-4B-reason-only.stride1-consumer.full source best-val/best snapshot of global_step_2176 kept by prestar/rl/tools/keep_best_val_checkpoint.py export tool prestar/rl/tools/export_hf_checkpoint.py (fp32, no cast) -->