qwen2.5-1.5b-ppo

qwen2.5-1.5b-ppo (research artifact)

A LoRA policy from PPO against qwen2.5-1.5b-rm-ultrafeedback, the classic two-step RLHF loop. Part of reward-model-ppo.

honest status

Research artifact, not an improved model. The PPO loop runs end to end with correct telemetry, but in this short, lightly-tuned run reward did not cleanly improve (it bounced -2 to -6 while KL to the reference drifted 12 -> 22). The value here is the characterization of PPO's two real costs - memory (it OOM'd on an 8 GB GPU and on an 80 GB A100 before being tuned to fit) and tuning instability - i.e. exactly what DPO was designed to avoid. Full teardown in the repo.

training

LoRA policy via TRL PPO (trl.experimental.ppo): policy + frozen reference (adapter-disabled)

• frozen reward model + trainable value head. batch 8, response length 64, kl_coef 0.05, 1,280 episodes, 1x A100.