agentic-rag-aerospace-grpo

Agentic RAG Aerospace — GRPO Fine-Tuned LoRA Adapter

A PEFT LoRA adapter for Qwen/Qwen2.5-0.5B-Instruct fine-tuned with Group Relative Policy Optimization (GRPO) on aerospace research RAG tasks. This model powers the Agentic RAG Gym — an RL environment for training AI agents to research like experts.

Quick Start — Usage with PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "williyam/agentic-rag-aerospace-grpo")
tokenizer = AutoTokenizer.from_pretrained("williyam/agentic-rag-aerospace-grpo")

messages = [
    {"role": "system", "content": "You are an aerospace research assistant. Analyze the retrieved documents and provide a comprehensive technical answer."},
    {"role": "user", "content": "Compare scramjet and ramjet propulsion for hypersonic vehicles based on the retrieved documents."}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Description

This adapter was trained as part of Agentic RAG Gym, an open-source reinforcement learning framework where autonomous agents learn to research through multi-step retrieval, reasoning, critique, and verification — supervised by real domain graders.

The GRPO training loop uses domain-expert graders as reward signals, not proxy rewards. The agents receive per-step feedback on retrieval relevance, reasoning quality, answer completeness, efficiency, and anti-reward-hacking penalties.

Used by Agentic RAG Gym HF Space

This model is used by the Agentic RAG Gym HF Space as the default fine-tuned model for aerospace domain tasks. The Space provides:

• Interactive Mode — Step through research tasks manually
• Auto Pilot — Watch the agent research autonomously
• Multi-Domain — Switch between Aerospace and Legal Research
• Real-Time Rewards — See per-step reward breakdowns

Training Details

Configuration

Reward Function

The composite reward combines five components (all bounded [0.01, 0.99]):

Anti-Reward-Hacking

Built-in adversarial guards penalize:

• Repetitive sentences/phrases
• Keyword stuffing without coherent reasoning
• Degenerate outputs (empty, trivially short, nonsensical)
• Query manipulation designed to game relevance scores

Evaluation Results

Task-Level Comparison

The model shows consistent improvement on easy and hard tasks (+5.3% to +5.6% on easy, +3.9% on hard), while medium-difficulty tasks remain stable. This pattern suggests the model learned domain-specific retrieval and reasoning strategies rather than shallow pattern matching.

Training Curves

Left: GRPO training loss over 80 steps — shows characteristic sharp drops during policy updates followed by stabilization.
Right: Mean reward (grader score) over training — fluctuates between 0.3-0.7 with an overall upward trend.

Baseline vs. GRPO-Trained

Side-by-side comparison across all 5 aerospace tasks. The GRPO-trained model (gold) outperforms or matches the baseline (dark red) on every task.

Score Distribution

The GRPO-trained model's score distribution shifts rightward compared to baseline, indicating more consistently higher scores.

Intended Use

This model is designed for agentic RAG tasks in aerospace research. It's trained to:

• Generate effective retrieval queries for aerospace topics
• Reason over technical documents (propulsion, materials, orbital mechanics)
• Produce structured, evidence-grounded technical answers
• Self-critique and refine its research strategy

Links

Citation

@misc{agentic-rag-gym-2026,
  title={Agentic RAG Gym: An RL Framework for Training AI Research Agents},
  author={williyam},
  year={2026},
  url={https://github.com/williyam-m/agentic-rag-gym}
}