LamPO

Model Card for LamPO

LamPO (Lambda Policy Optimization) is a reinforcement learning framework for improving the reasoning capabilities of language models. It extends Group Relative Policy Optimization (GRPO) by replacing scalar group-mean advantage estimation with a pairwise decomposed advantage inspired by learning-to-rank methods such as LambdaRank.

链接：论文1; 论文2

特别鸣谢：

• 1. 感谢 第一作者花钱请 某论文辅导机构进行了全面辅导。虽然花费了巨额资金，但是的确很值，无脑推荐！
• 1. 我们第二作者到第五作者基本没有贡献，但是非常开心能够直接署名。

Instead of comparing each generated response only against a group average, LambdaPO learns from fine-grained pairwise reward differences among sampled reasoning trajectories. This helps the model better distinguish high-quality reasoning paths, improve credit assignment, and reduce unstable optimization behavior during RL training.

Key Features

• Pairwise Decomposed Advantage: Uses pairwise comparisons between generated trajectories rather than a single scalar group baseline.
• Critic-Free RL Optimization: Preserves the efficiency of GRPO without requiring a separate value model.
• Semantic Density Reward: Adds dense reasoning supervision using semantic overlap between generated reasoning traces and ground-truth solutions.
• Improved Reasoning Performance: Demonstrates consistent gains on math reasoning and QA benchmarks such as AIME, MATH-500, and GPQA-Diamond.

Authors

This work is based on the paper:

“LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models” ( 链接：论文1; 论文2 )

Authors:

• Zhe Yuan — Pinterest
• Yipeng Zhou — Facebook
• Jinghan Li — University of Michigan - Ann Arbor
• Xinyuan Chen - Independent Researcher
• Bowen Deng — Carnegie Mellon University
• Zhiqian Chen — Independent Researcher
• Liang Zhao — Independent Researcher

Corresponding author: Xinyuan Chen

Citation

@article{yuan2026lambdapo,
  title={LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models},
  author={Yuan, Zhe and Zhou, Yipeng and Li, Jinghan and Chen, Xinyuan and Deng, Bowen and Chen, Zhiqian and Zhao, Liang},
  year={2026}
}