Downloads · 30 days
20
8% of all-time downloads
abhishekai/slm-125m-legal-sft
slm-125m-legal-sft is a text generation model from abhishekai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 125M-parameter Llama-architecture model, supervised fine-tuned from abhishekai/slm-125m-legal-base for grounded question answering over legal and financial passages.
Downloads · 30 days
20
8% of all-time downloads
All-time downloads
257
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors503 MB · 100%
From the Hugging Face model README
A 125M-parameter Llama-architecture model, supervised fine-tuned from
abhishekai/slm-125m-legal-base
for grounded question answering over legal and financial passages.
Unlike the base model (a text completer), this one follows instructions — but in a RAFT / retrieval-augmented style: it answers a question from a context passage you provide, which is the reliable behaviour a model this small can actually deliver. Remove the context and it has little to stand on.
The base model was fine-tuned on ~9,100 grounded Q&A pairs (plus an 800-pair held-out eval split). The data was generated by a teacher LLM (Gemini 2.5 Flash) reading passages sampled from the same legal/financial corpus the base model was pretrained on, then aggressively filtered:
Reused the base model's own 16K byte-level BPE tokenizer (no re-tokenization). A chat template is bundled and used at both training and inference.
Scored by a Claude Sonnet judge on four axes out of 10, over 300 held-out grounded Q&A prompts, on the same scale as every other checkpoint in this project.
| mean score | 4.05 / 10 |
| 95% CI | [3.73, 4.37] |
| paired step | +2.59 over the 125M pretrain base (1.46), CI [+2.26, +2.91] |
Per axis: QA 3.16, instruction following 3.27, grounding 4.24, refusal 5.52. Both RLAIF stages built on this checkpoint score no better — DPO 3.52 (a real regression) and PPO 4.08 (indistinguishable). At this scale the SFT is the model to use.
| Method | full-parameter SFT, loss on answer tokens only (prompt masked) |
| Examples | 9,096 train / 794 eval |
| Epochs | 3 (1,704 steps, micro-batch 16) |
| Optimizer | AdamW (beta 0.9/0.95) |
| LR schedule | 3% warmup to 2e-5, cosine decay to 2e-6 |
| Format | chat template; context folded into the user turn, RAFT-style |
| Hardware | 1x H100 |
| Compute cost | ~$0.32 (data generation ~$2.44 on top) |
Cross-entropy over answer tokens only on the held-out eval split:
| Metric | Value |
|---|---|
| Eval loss (answer tokens) | 1.213 |
| Eval perplexity (answer tokens) | 3.36 |
This is not comparable to the base model's 9.155 — that is general next-token perplexity over the whole sequence; this is loss over supervised answer tokens on grounded QA, an easier target. No downstream task benchmarks (LegalBench, CaseHOLD, etc.) have been run.
Give it a question and a context passage. Greedy decoding is strongly recommended — at higher temperatures a model this small drifts into the wrong task type.
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("abhishekai/slm-125m-legal-sft")
model = AutoModelForCausalLM.from_pretrained("abhishekai/slm-125m-legal-sft").eval()
question = "What standard of proof applies, and what must the plaintiff show?"
context = (
"In a civil negligence action the plaintiff must establish duty, breach, "
"causation, and damages. The standard of proof is a preponderance of the "
"evidence: the plaintiff must show it is more likely than not that the "
"defendant's conduct caused the harm."
)
messages = [
{"role": "system", "content": "You are a precise legal and financial assistant. "
"Answer using only the provided context. If the context does not contain the "
"answer, say you cannot answer from the context."},
{"role": "user", "content": f"{question}\n\nContext:\n{context}"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(text, add_special_tokens=False, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=160, do_sample=False,
eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
A research and educational artifact — a demonstration that a coherent, grounded instruction-following model can be distilled onto a 125M base for a few dollars. Not suitable for production, and specifically:
The model weights are released under Apache-2.0, inheriting the base model's license.
The fine-tuning data is synthetic, generated by Google's Gemini 2.5 Flash over passages from the base model's pretraining corpus (US case law — CC-BY-4.0; SEC filings — CC0-1.0; FineWeb-Edu — ODC-By 1.0). Synthetic data carries the teacher provider's usage terms; this artifact is published for research and education. If you intend to use it commercially, review Google's Gemini API terms and get your own counsel to look at it. None of this is legal advice.
The full fine-tuning pipeline — passage sampling, teacher generation, LLM-judge grounding, deduplication, task balancing, eval decontamination, tokenization, and the SFT run — runs on Modal and is reproducible end-to-end.