Downloads · 30 days
21
4% of all-time downloads
abhishekai/slm-125m-legal-ppo
slm-125m-legal-ppo is a text generation model from abhishekai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 125.8M-parameter grounded legal/financial Q&A model, tuned with PPO against a learned reward model on AI-generated feedback (RLAIF).
Downloads · 30 days
21
4% of all-time downloads
All-time downloads
470
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
A 125.8M-parameter grounded legal/financial Q&A model, tuned with PPO against a learned reward model on AI-generated feedback (RLAIF).
This is the only model in this project whose RLAIF stage scored above its SFT baseline — and the result is soft enough that you should not treat it as an upgrade.
| win-rate vs slm-125m-legal-sft | 0.570 |
| tally | 22 wins / 8 losses / 70 ties (n=100) |
| two-sided sign test on the 30 decisive pairs | p = 0.016 |
| 95% CI on the overall win-rate | [0.472, 0.663] — straddles 0.5 |
Three reasons to read that 0.570 conservatively:
Scored by Claude Sonnet on four axes out of 10 — a different model family from the Gemini judge behind the win-rate above, and one that had no hand in writing this project's training data. 300 held-out prompts, the same prompts and the same scale used for all twelve checkpoints, so this number is comparable across models in a way no win-rate here is.
| mean score | 4.08 / 10 |
| 95% CI (bootstrap over per-prompt scores) | [3.76, 4.41] |
| paired step | +0.03 against its own SFT (4.05), CI [-0.12, +0.19] - indistinguishable from the SFT. Note this disagrees with the 0.570 win-rate above; prefer this number, which is paired per prompt at n=300 and judged by a model that did not write the training data. |
The full twelve-checkpoint table, with every stage-to-stage interval, is in
MODEL_INDEX.md in the project repository.
Same pipeline, same prompt pool, same judge, same PPO code across all three models. PPO's win-rate falls with scale — the mirror image of DPO's rise:
| model | params | PPO win-rate | p | DPO win-rate |
|---|---|---|---|---|
| 125M (this) | 126M | 0.570 | 0.016 | 0.455 |
| 500M | 518M | 0.520 | 0.572 | 0.480 |
| Gemma-2-2B | 2,614M | 0.330 | 0.0002 | 0.530 |
Only the two ends are significant, and they point in opposite directions. The Gemma PPO checkpoint is a real degradation and is deliberately not published.
PPO maximizing reward - kl_coef * KL(policy ‖ SFT), 60 rollout steps, 32
prompts per rollout, kl_coef=0.2 on 1×H100. Realized: mean KL 0.080, final
KL 0.028, value loss ~0.005, no KL-stop abort, $0.17.
Three implementation details were load-bearing — without them the run is not merely worse, it is invalid:
kl_coef is only meaningful relative to reward
magnitude, and a Bradley-Terry reward model only fixes score differences —
its absolute scale is arbitrary. Scores are normalized per rollout batch.kl_coef. At step
1 the policy is the reference and KL ≈ 0, so a naive controller relaxes the
anchor exactly when it must hold.An earlier checkpoint trained before these fixes scored 0.48; it was overwritten and is not what is published here.
Trained from the same SFT backbone with a fresh scalar head and the pairwise Bradley-Terry loss, reaching 0.837 pairwise accuracy on held-out preference pairs. Published separately as slm-125m-legal-rm.
Identical to the SFT model — use the tokenizer's chat template; the assistant
turn ends on <|eos|>.
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-ppo")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-ppo")
msgs = [
{"role": "system", "content": "You are a precise legal and financial assistant. Answer only from the provided context."},
{"role": "user", "content": "Context: In a civil negligence action the plaintiff must prove duty, breach, causation, and damages by a preponderance of the evidence.\n\nQuestion: What standard of proof applies to the plaintiff?"},
]
text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors="pt", add_special_tokens=False),
max_new_tokens=128, do_sample=False)