slm-125m-sft

slm-125m-sft

A 126M-parameter model fine-tuned for grounded extractive question answering over US case law and SEC filings. Fine-tuned from prajwal55/slm-125m-base on 1x H100 in 3.1 minutes for $0.21.

Read this first

This model requires the source passage in the prompt. It answers questions about text you give it. It is not a closed-book model and has no useful recall of legal facts on its own -- ask it a question without a passage and it will invent an answer.

On held-out data it produces a substantially correct answer about 60% of the time and a clearly wrong one about 10% of the time. It is a demonstration of an end-to-end from-scratch pipeline at small scale, not a system to rely on.

Usage

The chat format uses dedicated special tokens. There is no chat_template; build the prompt directly:

from transformers import AutoTokenizer, LlamaForCausalLM
import torch

tok = AutoTokenizer.from_pretrained("prajwal55/slm-125m-sft")
model = LlamaForCausalLM.from_pretrained("prajwal55/slm-125m-sft",
                                         torch_dtype=torch.bfloat16).eval()

passage = "..."   # the text the answer must come from
question = "..."
system = "You are a legal and financial assistant. Answer the question using only the passage provided. Be precise and concise."

prompt = (f"<|bos|><|system|>{system}<|user|>{passage}\n\n"
          f"Question: {question}<|assistant|>")
ids = tok(prompt, add_special_tokens=False, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=160, do_sample=False,
                     eos_token_id=tok.convert_tokens_to_ids("<|eos|>"))
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))

Passage plus question plus answer must fit the 1024-token context. Passages of ~2,400 characters leave comfortable room.

Training data

8,000 grounded QA pairs distilled from deepseek/deepseek-v4-flash over passages sampled from HFforLegal/case-law and PleIAs/SEC.

Each pair passed a QC gauntlet: the teacher had to cite a verbatim supporting span, then length/format filters, embedding-based near-duplicate question clustering, an LLM grounding judge, and a 13-gram decontamination re-check against CaseHOLD applied to the generated text as well as the source passage. 15,894 raw pairs were filtered to 8,000.

Train and validation are disjoint by source passage, not merely by pair -- each passage yields ~3.6 questions, so a pair-level split would put the same passage on both sides.

Training

For comparison, the base model's pretraining was 8x H100, ~28 min, ~$18.

The epoch count was chosen empirically: 1-, 2-, and 3-epoch runs were each trained with a schedule fully annealed at their own final step, then compared. 3 epochs overfit (val loss 0.4981 vs 0.4665) while scoring identically on answer correctness; 1 epoch was worse on both.

Results

Held-out validation, 1,003 examples, greedy decoding:

On the perplexity number: 1.6 looks excellent and is not the achievement it appears to be. The fine-tuning data is ~55% extractive -- the teacher was required to quote a supporting span -- so predicting the answer given the passage is largely a copying task. Token F1 is the honest metric here.

On CaseHOLD: unchanged from the base model, as expected. CaseHOLD is 5-way multiple-choice holding selection; this model was tuned for extractive QA. The task does not transfer and both numbers are at chance.

Limitations

• Requires the source passage. No useful closed-book knowledge.
• ~10% of answers are clearly wrong. Failures include answering a different question than asked, and occasional digit errors when copying figures (3% of numeric answers contain an ungrounded number).
• Answers reflect the teacher model's phrasing and biases; all training data is synthetic, from a single teacher.
• English only, 1024-token context, case-law inputs carry OCR noise.
• Generation is somewhat sensitive to batching and padding.
• Not legal or financial advice. Unsuitable for any use where an incorrect statement of law or fact carries consequences.