slm125m-instruct

slm125m-instruct

A 126M-parameter legal/financial Q&A model, supervised fine-tuned from nitin1pawar/slm125m-base — a base model that was itself pretrained from scratch (corpus, tokenizer and weights) for $31.76.

Trained in RAFT style: each example pairs a question with retrieved passages, and a quarter of the training set contains passages that do not answer the question, so the model learns to say so instead of inventing an answer.

Prompt format

<|bos|><|system|>{system}<|user|>{user}<|assistant|>{answer}<|eos|>

tokenizer.apply_chat_template is configured, so:

messages = [
    {"role": "system", "content": "You are a legal and financial assistant. ..."},
    {"role": "user", "content": "Passage 1:\n...\n\nQuestion: ..."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

When the passages do not contain the answer the model replies:

The passage does not provide enough information to answer this question.

Training data

10,000 synthetic Q&A pairs generated with gpt-5-mini from the model's own pretraining corpus (US case law + SEC filings + educational web text), then graded by an LLM judge on groundedness, correctness, answerability and form. Judge pass rate 88.4%. Questions were deduplicated by exact hash and by 5-gram Jaccard overlap.

Passages were drawn from the deduplicated and decontaminated corpus, so CaseHOLD and LexGLUE case_hold remain genuinely held out.

Training

148 steps over 1 epochs (9.7M tokens, 62s on 1×H100), full-parameter (no LoRA), AdamW, peak LR 8e-05 cosine to 0, bf16. Loss is computed on answer tokens only — question and context tokens are masked out.

Results

Limitations

125M parameters. It produces fluent but frequently incorrect legal and financial text, its knowledge is bounded by a 2.07B-token corpus, and the training answers were written by another language model rather than by lawyers. Context is capped at 1,024 tokens. Not legal or financial advice.