DGPO-qwen2.5-0.5b

<div align="center"> <!-- # 🔍 DGPO: Distillation-Guided Policy Optimization -->

Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities

<p align="center"> <a href="https://arxiv.org/abs/2508.20324"><img src="https://img.shields.io/badge/Paper-arXiv-red?logo=arxiv"></a> <!-- <a href="#"><img src="https://img.shields.io/badge/GitHub-Code-black?logo=github"></a> --> </p> <!-- *DGPO enables compact language models (0.5–1B) to perform agentic RAG with stable reinforcement learning.* --> </div>

---

🧠 Overview

DGPO (Distillation-Guided Policy Optimization) is a reinforcement learning framework with integrated knowledge distillation, designed to enable agentic search behaviors in compact language models.

While RL works well for large models, compact models suffer from:

• ❌ Poor initial outputs
• ❌ Training collapse in RL
• ❌ Ineffective exploration

DGPO solves this by combining:

• ✅ Cold-start knowledge distillation (KD)
• ✅ Teacher-guided reinforcement learning

This enables stable learning and even allows compact models to match or surpass teacher models

---

⚙️ Key Idea

🔁 Distillation-Guided RL

DGPO introduces a simple but powerful principle:

✅ Reward if correct ❌ Mimic teacher if wrong

This creates a stable learning signal even when the model is weak.

---

🏗️ Framework

1. Cold-Start Initialization (KD)

• Train student using teacher-generated outputs (TGO)
• Provides high-quality trajectories
• Prevents early collapse

2. Distillation-Guided RL

• Use PPO-based RL
• Reward correct answers
• Apply selective KL penalty only when wrong

This enables:

• Stable training
• Efficient exploration
• Error-focused learning

---

🔍 Agentic RAG Behavior

DGPO trains models to perform multi-step search reasoning:

<think> reasoning </think>

<search> query </search>  
<information> retrieved docs </information>  
<answer> final answer </answer>  

---

🚀 Performance

Overall QA Performance

📊 Qwen2.5 (3B → 0.5B)

👉 DGPO achieves ~55× improvement over base model

👉 In some cases, student surpasses teacher

---

🎓 Citation

@article{kotoge2025dgpo,
title={Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization},
author={Kotoge, Rikuto and Nishimura, Mai and Ma, Jiaxin},
journal={arXiv preprint arXiv:2508.20324},
year={2025}
}

@inproceedings{
kotoge2025democratizing,
title={Democratizing Agentic {RAG}: Distillation-Guided Policy Optimization  for Compact Language Models},
author={Rikuto Kotoge and Mai Nishimura and Jiaxin Ma},
booktitle={NeurIPS 2025 Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning},
year={2025},
url={https://openreview.net/forum?id=CP0H9NAWES}
}

---

🤝 Acknowledgements

• Search-R1 (agentic RL baseline)
• Open models (Qwen & Llama)
• Open QA benchmarks (NQ, HotpotQA, etc.)

---

<div align="center">

DGPO makes agentic RAG accessible to small models 🚀

</div>