Downloads · 30 days
14
3% of all-time downloads
abhishekai/slm-125m-legal-dpo-v2
slm-125m-legal-dpo-v2 is a text generation model from abhishekai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
Direct Preference Optimization on top of abhishekai/slm-125m-legal-sft-v2, for grounded question answering over legal and financial passages.
Downloads · 30 days
14
3% of all-time downloads
All-time downloads
451
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
Direct Preference Optimization on top of
abhishekai/slm-125m-legal-sft-v2,
for grounded question answering over legal and financial passages.
This is the best 125M checkpoint in the project, and the first RLAIF result here that an independent judge confirms rather than reverses.
| mean /10 | qa | instr. following | grounding | refusal | |
|---|---|---|---|---|---|
sft-v2 (its own baseline) | 4.13 | 3.18 | 3.29 | 4.52 | 5.55 |
dpo-v2 | 4.82 [4.48, 5.17] | 3.85 | 3.77 | 5.71 | 5.96 |
Paired over the same 300 prompts: +0.69, bootstrap 95% CI [+0.34, +1.04], 156 W / 112 L / 32 T. The interval excludes zero, so this is a measured improvement, not a point estimate that happens to be positive. Every axis rose; grounding most (+1.19).
Repetition collapse also more than halved, 4.3% → 1.7% — which matters more for a served model than a judged one.
Two defects made every previously published DPO/PPO number here uninformative, and both are fixed in this checkpoint:
sft-v2 itself, the model being improved.The funnel: 1,194 prompts → 758 ranked pairs → 725 confirmed by an independent pairwise check → 625 train / 100 eval.
Its Gemini-judged win-rate is 0.545 at n=100, which is not significant (p = 0.349). That number is not the evidence for this model, and the reason is worth stating plainly: Gemini ranked the preference pairs and judged the win-rate, so it was asked whether a policy trained toward its own taste matched its own taste.
Scored instead by Claude Sonnet — a different model family, paired over 3× the prompts on an absolute scale — the same comparison gives +0.69 with an interval excluding zero. On the identical run, Gemini scored a Gemma DPO checkpoint at the same 0.545 while Sonnet found it had regressed by −0.97. Two identical-looking win-rates, opposite outcomes.
Prefer the absolute scores above to any win-rate, including this model's own.
DPO, 114 steps, held-out DPO loss 0.6716, preference accuracy 0.865, margin 0.654. 1×H100, 5.8 minutes, $0.38. Best-checkpoint selection is on held-out DPO loss, not preference accuracy — an earlier run reached 0.96 accuracy at a 0.455 win-rate, so accuracy is not the quantity that tracks quality here.
It did not converge. The best evaluation was the final step and early stopping never fired, so 0.6716 is a floor for this recipe rather than its ceiling.
It expects a chat template with a system prompt, and a user turn containing the question
followed by Context: and the passage. Use tokenizer.apply_chat_template.
Claude Sonnet as an independent judge — a different model family from the Gemini 2.5 Flash that wrote the training data and ranked the preference pairs. Four axes out of 10 (question answering, instruction following, grounding in the supplied passage, appropriate refusal), scored against the prompt, the answer, a golden answer and the corpus evidence. The same 300 held-out prompts score every model in this project.
The judge was gated before use: it had to reproduce a known ordering (base Gemma-2-2B above a broken fine-tune of it) before any of its scores were trusted. It did, at p = 0.0022.