Downloads · 30 days
16
4% of all-time downloads
abhishekai/slm-500m-legal-ppo
slm-500m-legal-ppo is a text generation model from abhishekai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 517.8M-parameter grounded legal/financial Q&A model, tuned with PPO against a learned reward model on AI-generated feedback (RLAIF).
Downloads · 30 days
16
4% of all-time downloads
All-time downloads
433
Public
Parameters
518M
2.1 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2.1 GB · 100%
From the Hugging Face model README
A 517.8M-parameter grounded legal/financial Q&A model, tuned with PPO against a learned reward model on AI-generated feedback (RLAIF).
This model performs at parity with the SFT model it was trained from.
| win-rate vs slm-500m-legal-sft | 0.520 |
| tally | 16 wins / 12 losses / 72 ties (n=100) |
| two-sided sign test on the 28 decisive pairs | p = 0.572 |
| 95% CI on the overall win-rate | [0.423, 0.615] |
The interval straddles 0.5. PPO here did not measurably improve the model and did not measurably harm it. If you want the better-understood model, use the SFT one.
The 72 ties are the substantive result, not a footnote. Mean KL from the SFT reference across training was 0.134 — the policy barely moved, so the judge mostly cannot distinguish the two models. A numerically healthy PPO run at this scale produces nearly the same model.
Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.
| mean score | 4.51 / 10 |
| 95% CI (bootstrap over per-prompt scores) | [4.19, 4.84] |
| paired step | -0.15 against its own SFT (4.65), CI [-0.35, +0.06] - indistinguishable from the SFT. |
The full twelve-checkpoint table, with every stage-to-stage interval, is in
MODEL_INDEX.md in the project repository.
Same pipeline, same prompt pool, same Gemini 2.5 Flash judge, same PPO code across all three models. PPO's win-rate falls with scale — the mirror image of DPO's rise:
| model | params | PPO win-rate | p | DPO win-rate |
|---|---|---|---|---|
| 125M | 126M | 0.570 | 0.016 | 0.455 |
| 500M (this) | 518M | 0.520 | 0.572 | 0.480 |
| Gemma-2-2B | 2,614M | 0.330 | 0.0002 | 0.530 |
Only the two ends are significant, and they point in opposite directions. The Gemma PPO checkpoint is a real degradation and is deliberately not published.
The reward model distilled Gemini 2.5 Flash's preferences, PPO optimized against that reward model, and Gemini 2.5 Flash then judged the result. Win-rates under those conditions partly measure agreement with the judge's own taste. An independent judge would be needed to break that circularity.
PPO maximizing reward - kl_coef * KL(policy ‖ SFT), 60 rollout steps, 16 prompts
per rollout, kl_coef=0.2 on 1×H100. Realized: mean KL 0.134, final KL 0.556,
value loss 0.0005–0.035, kl_coef never left its floor, no abort, 13.3 min,
$0.23.
Three implementation details were load-bearing — without them the run is not merely worse, it is invalid:
kl_coef is only meaningful relative to reward
magnitude, and a Bradley-Terry reward model only fixes score differences —
its absolute scale is arbitrary. Scores are normalized per rollout batch.kl_coef, since
at step 1 the policy is the reference and KL ≈ 0.Trained from the same SFT backbone with a fresh scalar head and the pairwise Bradley-Terry loss, reaching 0.820 pairwise accuracy on held-out preference pairs. Published separately as slm-500m-legal-rm.
Use the chat template shipped with the tokenizer. The assistant turn ends on
<|eos|>.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("abhishekai/slm-500m-legal-ppo")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-500m-legal-ppo")
msgs = [
{"role": "system", "content": "You are a precise legal and financial assistant. Answer only from the provided context."},
{"role": "user", "content": "Context: In a civil negligence action the plaintiff must prove duty, breach, causation, and damages by a preponderance of the evidence.\n\nQuestion: What standard of proof applies to the plaintiff?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False),
max_new_tokens=128, do_sample=False)