slm125m-live-sft

slm125m-live-sft

The 125.8M-parameter legal/financial model AnandHaridas1980/slm125m-live after supervised fine-tuning on 2,620 grounded question-answer pairs.

The base model continues text. This one answers a question from a passage you supply, or says the passage does not contain the answer.

What changed

Measured on 200 held-out pairs never seen in training, greedy decoding, judged by gemini-3.6-flash against the source passage.

Accuracy by question type:

Read this before using it

The 36% headline is carried by refusals. On questions that genuinely have an answer in the passage it is right about 21% of the time. On questions that do not, it is right 86.7% of the time.

In plain terms: this model learned when not to answer far better than how to answer. It reliably produces a well-formed, correctly terminated, confident-sounding response — and that response is frequently wrong on specifics. Every failure looks like a competent answer.

Verify every figure, date and name against the passage. Do not use this as an answer service.

It is useful as a component: a cheap first-pass reader whose refusals are trustworthy enough to route on. It is not useful as an authority.

Prompt format (required)

The model was trained on exactly one prompt shape. Deviating from it degrades output silently.

<|bos|><|system|>You are a legal and financial assistant.
Answer only from the provided context.
If the context is not enough, say you do not know.<|user|>Context:
{passage}

Question: {question}<|assistant|>

Generate from there; the model emits the answer then <|eos|>.

import torch
from transformers import AutoTokenizer, LlamaForCausalLM

tok = AutoTokenizer.from_pretrained("AnandHaridas1980/slm125m-live-sft")
model = LlamaForCausalLM.from_pretrained("AnandHaridas1980/slm125m-live-sft", torch_dtype=torch.bfloat16).eval()
model.config.use_cache = True          # ships False from training; 6x slower without it

SYSTEM = ("You are a legal and financial assistant.\n"
          "Answer only from the provided context.\n"
          "If the context is not enough, say you do not know.")

def ask(passage, question, max_new_tokens=128):
    prompt = (f"<|bos|><|system|>{SYSTEM}<|user|>Context:\n{passage}\n\n"
              f"Question: {question}<|assistant|>")
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).input_ids
    out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False,
                         eos_token_id=tok.convert_tokens_to_ids("<|eos|>"),
                         pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))
    return tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True).strip()

print(ask("Net revenue rose 12.4% to $48,300,000 in fiscal 2025.",
          "What was net revenue in fiscal 2025?"))

With no passage it should refuse. That is trained behaviour, not a bug — it has no useful world knowledge and was never meant to.

Training

Validation loss bottomed at step 80 (1.1143) and drifted to 1.1449 by step 120; the published checkpoint is step 120. Two epochs would likely have been better.

Limitations

• Not RAFT. Every training example contained exactly one passage, always the correct one. The model has never seen an irrelevant passage, so behind a real retriever returning mixed chunks it is out of distribution.
• Context limit 1,024 tokens, including the passage.
• Single turn only. No multi-turn conversation was trained.
• English only; US case law and SEC filings.
• Reasoning accuracy is 9.6%. Do not use it for multi-step inference.
• Judged by a model from the same family that generated the training data, which is a weaker check than an independent evaluator.

Full write-up

The complete build — dataset construction, cost model, every failure — is documented in doc/sft/ of the project repository, alongside the pretraining book for the base model.