ExGRPO-Llama3.1-8B-Instruct

ExGRPO-Llama3.1-8B-Zero: Learning to Reason from Experience

This repository hosts the ExGRPO-Llama3.1-8B-Zero model, a component of the ExGRPO: Learning to Reason from Experience framework. This work was presented in the paper:

ExGRPO: Learning to Reason from Experience

Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. ExGRPO (Experiential Group Relative Policy Optimization) addresses the computational inefficiency and instability of standard on-policy training by investigating the value of reasoning experiences and proposing a framework that organizes and prioritizes valuable experiences. It employs a mixed-policy objective to balance exploration with experience exploitation, leading to consistent improvements in reasoning performance on mathematical and general benchmarks.

For the official code and further details on the ExGRPO framework, please visit the project's GitHub repository: GitHub Repository

<div align="center"> <img src="https://github.com/ElliottYan/LUFFY/raw/main/ExGRPO/figures/exgrpo_intro.png" alt="overview" style="width: 88%; height: auto;"> </div>

Key Highlights

ExGRPO introduces significant advancements in RLVR for reasoning tasks:

• Experience Value Modeling: Introduces online proxy metrics (rollout correctness and trajectory entropy) for quantifying the value of RLVR experience.
• ExGRPO Framework: Built on top of GRPO, ExGRPO introduces a systematic experience management mechanism and an experience optimization objective to maximize the benefit of past explorations.
• Generalization and Stability: Demonstrates broad applicability across different backbone models and mitigates training collapse of on-policy RLVR in challenging scenarios.

Getting Started & Usage

This model is designed to be compatible with the Hugging Face transformers library. A default sample usage snippet will be automatically displayed on the Hugging Face Hub page.

To set up the environment and explore the project, follow the installation instructions from the GitHub repository:

Installation

You can install dependencies by running the following commands:

conda create -n exgrpo python=3.10
conda activate exgrpo
cd exgrpo
pip install -r requirements.txt
pip install -e .
cd verl
pip install -e .

Note: If you encounter issues caused by the pyairports library, please refer to this hot-fix solution.

For the flash-attn library, we use the v2.7.4-post1 release and recommend installing it via the pre-built wheel. Please adjust based on your environment.

wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
pip install flash_attn-2.7.4.post1+cu12torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

Data Preparation and Training

For data preparation and training instructions, please refer to the Usage section in the GitHub repository.

Evaluation

The GitHub repository also provides scripts for evaluating the model's performance on various benchmarks. For example, you can use the generate_vllm.py script for generation and evaluation, as detailed in the Evaluation section of the GitHub repository.

Released Models

The ExGRPO framework includes several released models. This table (copied from the GitHub README) provides an overview:

Citation

If you find our model, data, or evaluation code useful, please kindly cite our paper:

@article{zhan2025exgrpo,
      title={ExGRPO: Learning to Reason from Experience}, 
      author={Runzhe Zhan and Yafu Li and Zhi Wang and Xiaoye Qu and Dongrui Liu and Jing Shao and Derek F. Wong and Yu Cheng},
      year={2025},
      journal = {ArXiv preprint},
      volume = {2510.02245},
      url={https://arxiv.org/abs/2510.02245}, 
}