PPO-step-2

On Predictability of Reinforcement Learning Dynamics for Large Language Models

This repository provides one of the models used in our paper "On Predictability of Reinforcement Learning Dynamics for Large Language Models" for evaluating and predicting reinforcement learning (RL) dynamics in large language models (LLMs).

Recent advances in LLM reasoning capabilities are largely driven by RL, yet the parameter dynamics during RL training remain poorly understood. Our work identifies two key properties of RL-induced parameter updates: Rank-1 Dominance, where the top singular subspace of the parameter update matrix captures nearly all reasoning improvements, and Rank-1 Linear Dynamics, where this subspace evolves linearly across training, allowing accurate prediction from early checkpoints. Based on these insights, we propose AlphaRL, a plug-in acceleration framework that extrapolates final parameter updates from a short early training window, achieving up to 2.5× speedup while retaining over 96% of reasoning performance.

This model is one of the training checkpoints used in our paper and is provided to support research on evaluating and predicting parameter dynamics during RL training of LLMs. The full codebase is available at: AlphaRL GitHub.

🔧 Prompt Format (Chat Template)

During Inference, each question is formatted as:

{question} Please reason step by step, and put your final answer within boxed{}.

Then wrapped using the chat template:

prompt = tokenizer.apply_chat_template(
    [{{"content": question_with_instruction, "role": "user"}}],
    tokenize=False,
    add_generation_prompt=True,
)

🧪 Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("caiyuchen/PPO-step-2")
tokenizer = AutoTokenizer.from_pretrained("caiyuchen/PPO-step-2")

question = "Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $(r,\theta),$ where $r > 0$ and $0 \le \theta < 2 \pi.$"
question_with_instruction = question + "Please reason step by step, and put your final answer within \boxed{{}}"

# Apply chat template
prompt = tokenizer.apply_chat_template(
    [{{"content": question_with_instruction, "role": "user"}}],
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

📎 Reference

If you find this model useful, please consider citing our paper:

🔗 Paper Link: https://huggingface.co/papers/2510.00553

@misc{cai2025predictabilityreinforcementlearningdynamics,
      title={On Predictability of Reinforcement Learning Dynamics for Large Language Models}, 
      author={Yuchen Cai and Ding Cao and Xin Xu and Zijun Yao and Yuqing Huang and Zhenyu Tan and Benyi Zhang and Guiquan Liu and Junfeng Fang},
      year={2025},
      eprint={2510.00553},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2510.00553}, 
}