slm125LIVE-sft

slm125LIVE-sft

A 125M legal/financial SLM fine-tuned from thesreedath/slm-125m-base on grounded (RAFT-style) question-answer data. Each example supplies a context passage; the model answers from that context, or refuses when the answer is not present.

• Objective: supervised fine-tuning, loss on answer tokens only.
• Data: 12000 Q&A pairs generated by gpt-5-mini from a cleaned US case-law / SEC / web corpus, then filtered by a verbatim-evidence check, an LLM judge (gpt-5-nano), near-duplicate removal and decontamination against CaseHOLD / LexGLUE.
• Training: 3 epochs on 1xH100, 36.9M tokens seen.
• Held-out loss: 4.3636 (base) -> 0.8397 (tuned).

Prompt format

prompt = ("<|bos|><|system|>
{system}<|eos|>
<|user|>
<context>
"
          "{PASSAGE}
</context>

Question: {QUESTION}<|eos|>
<|assistant|>
")

Where {system} is: You are a legal and financial assistant. Answer the question using ONLY the provided context.

When the context does not contain the answer the model replies: The provided context does not state this.

Note on special tokens

The base repo's config.json declares bos_token_id: 1, eos_token_id: 2, which disagrees with its own tokenizer.json (<|bos|>=0, <|eos|>=1, <|pad|>=2). The tokenizer is authoritative -- the model was pretrained emitting ID 1 at document boundaries. This repo ships the corrected IDs, so generation stops on a real <|eos|> instead of running to the token limit.