DemyAgent-4B

<div align="center"> <h1>Demystifying Reinforcement Learning in Agentic Reasoning<h1> <p align="center"> <a href="https://arxiv.org/abs/2510.11701"> <img src="https://img.shields.io/badge/Paper-Arxiv-red?logo=arxiv&logoColor=red" alt="Paper on arXiv"/> </a> <a href="https://github.com/Gen-Verse/Open-AgentRL"> <img src="https://img.shields.io/badge/Open--AgentRL-GitHub-black?logo=github&logoColor=white" alt="Open-AgentRL on GitHub"/> </a> <a href="https://huggingface.co/datasets/Gen-Verse/Open-AgentRL-30K"> <img src="https://img.shields.io/badge/30K_RL_Dataset-Hugging%20Face-orange?logo=huggingface&logoColor=yellow" alt="30K RL Dataset"/> </a> <a href="https://huggingface.co/Gen-Verse/DemyAgent-4B"> <img src="https://img.shields.io/badge/DemyAgent--4B-Hugging%20Face-FFCC00?logo=huggingface&logoColor=yellow" alt="DemyAgent-4B Model"/> </a> </p> </div>

🎯 About This Repository

This repository contains the DemyAgent-4B model weights, a 4B-sized agentic reasoning model that achieves state-of-the-art performance on challenging benchmarks including AIME2024/2025, GPQA-Diamond, and LiveCodeBench-v6. DemyAgent-4B is trained using our GRPO-TCR recipe with 30K high-quality agentic RL data, demonstrating that small models can outperform much larger alternatives (14B/32B) through effective RL training strategies.

🌟 Introduction

In our work, we systematically investigate three dimensions of agentic RL: data, algorithms, and reasoning modes. Our findings reveal:

• 🎯 Data Quality Matters: Real end-to-end trajectories and high-diversity datasets significantly outperform synthetic alternatives
• ⚡ Training Efficiency: Exploration-friendly techniques like reward clipping and entropy maintenance boost training efficiency
• 🧠 Reasoning Strategy: Deliberative reasoning with selective tool calls surpasses frequent invocation or verbose self-reasoning We contribute high-quality SFT and RL datasets, demonstrating that simple recipes enable even 4B models to outperform 32B models on the most challenging reasoning benchmarks.

📦 Resources

Note:

• Qwen2.5-7B-RA-SFT and Qwen3-4B-RA-SFT are finetuned from Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 using our 3K Agentic SFT Data
• DemyAgent-4B is trained through Agentic RL with our 30K Agentic RL data using the GRPO-TCR recipe

🏆 Performance

We evaluate our models on challenging benchmarks spanning mathematics, science, and code generation tasks.

Benchmark Results

Key Highlights

✨ Despite having only 4B parameters, DemyAgent-4B achieves:

• 🥇 State-of-the-art on AIME2025 (70.0%), outperforming even DeepSeek-R1-Zero (671B)
• 🥈 Second place on AIME2024 (72.6%) and GPQA-Diamond (58.5%)
• 🚀 Competitive performance against 14B-32B models with 4-8× fewer parameters
• 💡 Superior efficiency compared to long-CoT models through deliberative tool use

📝 Citation

@article{yu2025demystify,
  title={Demystifying Reinforcement Learning in Agentic Reasoning},
  author={Yu, Zhaochen and Yang, Ling and Zou, Jiaru and Yan, Shuicheng and Wang, Mengdi},
  journal={arXiv preprint arXiv:2510.11701},
  year={2025}
}