slm-125m-sft

slm-125m-sft

A 125M legal SLM fine-tuned (SFT) from thesreedath/slm-125m-base (the 10-epoch base) on grounded (RAFT-style) question-answer data: each example provides a context passage and a question, and the model answers from the context (or refuses when the answer is not present).

• Objective: supervised fine-tuning, loss on answer tokens only.
• Data: ~15k Gemini-generated Q&A pairs grounded in the cleaned case-law / SEC corpus, cleaned (faithfulness filter + dedup + decontamination vs CaseHOLD/LexGLUE).
• Use it with a context passage for reliable, honest answers (pairs with retrieval).

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("thesreedath/slm-125m-sft")
model = AutoModelForCausalLM.from_pretrained("thesreedath/slm-125m-sft")
prompt = ("<|bos|><|system|>\nYou are a legal and financial assistant. Answer the "
  "question using ONLY the provided context.<|eos|>\n<|user|>\n<context>\n"
  "{PASSAGE}\n</context>\n\nQuestion: {QUESTION}<|eos|>\n<|assistant|>\n")

Full pipeline + code: https://github.com/Vizuara-AI-Lab/slm-125m-from-scratch