SLM-RL-Agents

SLM-RL-Agents — Model Checkpoints

Paper: Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

Authors: Md Rezwanul Haque, Md. Milon Islam, Fakhri Karray

Paper: arXiv:2607.25091

Code: github.com/rezwanh001/slm-rl-agents

This repository hosts 30 trained checkpoints (15 SFT + 15 PPO) from the SLM-RL-Agents framework — a stabilised RLHF pipeline for training small language model agents in the 70M–410M parameter regime — plus an agentic-SFT warm-up checkpoint released as forward-compatibility scaffolding for the multi-turn agentic extension.

All numerical claims below match the corresponding tables of the paper and are reproducible from results/all_results.json in the code repository.

Models

Corpora

• TinyStories — simple narrative fiction
• CNN/DailyMail — news articles
• Wikitext-103 — encyclopaedic text

Repository Layout

SLM-RL-Agents/
├── sft/                    # 15 LoRA adapters
│   ├── pythia-70m/{tinystories, cnn_dailymail, wikitext}/
│   ├── pythia-160m/...
│   ├── pythia-410m/...
│   ├── smollm2-135m/...
│   └── smollm2-360m/...
├── ppo/                    # 15 fully merged models
│   ├── pythia-70m/{tinystories, cnn_dailymail, wikitext}/
│   └── ...
└── agentic_sft/            # forward-compatibility tool-use warm-up
    └── pythia-410m/task_a_tinystories/   # full FT, tool-call acc 1.000

Agentic-SFT Warm-up (forward-compatibility)

A single tool-use SFT checkpoint released alongside the multi-turn agentic scaffolding in src/agentic/ and the corresponding dataset (agentic/task_a_tinystories/ in SLM-RL-Agents-Data). This checkpoint teaches a base SLM the action-sentinel grammar (<tool name="X">{...}</tool>, <ask>...</ask>, <finish>...</finish>) required by src.agentic.environment.AgenticEnvironment.

Status: Released as scaffolding only — not evaluated empirically in the paper. Smaller variants (pythia-70m, pythia-160m) with LoRA failed the ≥0.5 precondition; pythia-410m full FT was required to clear it.

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

root = snapshot_download(
    repo_id="mr3haque/SLM-RL-Agents",
    allow_patterns="agentic_sft/pythia-410m/task_a_tinystories/**",
)
path = f"{root}/agentic_sft/pythia-410m/task_a_tinystories"
model = AutoModelForCausalLM.from_pretrained(path)
tok = AutoTokenizer.from_pretrained(path)

Key Results — Five Configurations Where PPO Helps

The paper's central finding (capacity-headroom hypothesis) is that PPO yields a positive reward delta only where the SFT prior is fluent (PPL < 20) and the reward signal is informative. Across the 15 configurations, exactly five rows clear that bar:

Reward scores use per-configuration scales (each reward model is trained from the matching SFT checkpoint, so absolute magnitudes are not comparable across rows). The remaining 10 configurations train stably without divergence but show near-zero or negative deltas — consistent with the capacity-headroom prediction. The full 15-row table (including all PPL, reward, ROUGE, BLEU, and Distinct-N values) is in Table II/V of the paper and in results/all_results.json in the code repository.

Statistical significance. A two-sided z-test over the n = 200 held-out prompts per configuration (95% CI on Δ) confirms that three of these gains are statistically significant — Pythia-410M / TinyStories (CI [+0.61, +2.10], p < 0.001), SmolLM2-360M / TinyStories (CI [+0.32, +1.13], p < 0.001), and SmolLM2-360M / Wikitext-103 (CI [+0.04, +0.50], p < 0.05); the two remaining positive deltas lie within sampling noise. The only significant regression, Pythia-410M / Wikitext-103 (Δ = −1.043, p < 0.001), coincides with an SFT prior above the fluency threshold (PPL = 25.4), in agreement with the capacity-headroom hypothesis.

Quick Start

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

root = snapshot_download(repo_id="mr3haque/SLM-RL-Agents", allow_patterns="ppo/smollm2-360m/tinystories/**")
model = AutoModelForCausalLM.from_pretrained(f"{root}/ppo/smollm2-360m/tinystories")
tokenizer = AutoTokenizer.from_pretrained(f"{root}/ppo/smollm2-360m/tinystories")

Datasets

mr3haque/SLM-RL-Agents-Data

Citation

@inproceedings{haque2026slmrlagents,
  title         = {Towards Robust Reinforcement Learning for Small-Scale
                   Language Model Agents},
  author        = {Haque, Md Rezwanul and Islam, Md. Milon and Karray, Fakhri},
  booktitle     = {Proceedings of the IEEE International Conference on
                   Systems, Man, and Cybernetics (SMC)},
  year          = {2026},
  eprint        = {2607.25091},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2607.25091},
  note          = {University of Waterloo \& KUET \& MBZUAI}
}