slm-125m-raft-finetuned

slm-125m-raft-finetuned

RAFT (Retrieval-Augmented Fine-Tuning) SFT of Probabilistr/slm-125m-base (a 125.8M-parameter from-scratch Llama-architecture model) -- unlike Probabilistr/slm-125m-finetuned (single gold-passage SFT), this model is trained on 5 retrieved documents per example (80% include the oracle passage among distractors, the rest are all-distractor "negative" examples trained to answer "Not stated in the context."), so it learns to identify the right passage among retrieval noise and to refuse when none of the retrieved documents actually answer the question.

Fine-tuning data

~7,500 examples built from the same case-law/SEC-filing Q&A pool as slm-125m-finetuned (grounded QA/RAFT, summarization, extraction-to-JSON, and plain-English rewrite tasks generated with gpt-4o-mini), reassembled into RAFT's multi-document retrieval format with distractors sampled from embedding nearest-neighbors. See RAFT_instructions.txt and raft_topup_data.py/raft_assemble.py for the full pipeline.

Training

• 1x A10G, 3 epochs, 1389 steps
• Passages truncated to 140 tokens each (5 docs must fit the 1,024-token context)
• Loss masked to only the assistant-response tokens

Known limitations

Passages are truncated much shorter than the single-passage model's (140 vs 450 tokens) to fit 5 documents in a 1,024-token context -- this small model may lose some detail that would be visible with more context. See metrics.jsonl / summary.json in this repo for the training curve.

Prompt format

<|bos|><|system|>
You are a legal/financial assistant. Answer strictly using the information in the provided context. If the answer is not stated in the context, say "Not stated in the context."<|user|>
Context:
[Document 1]
{doc_1}

[Document 2]
{doc_2}
...

Instruction: {instruction}<|assistant|>