slm-125m-finetuned-qa

slm-125m-finetuned-qa

A 125.8M-parameter LLaMA-architecture model, fine-tuned from thesreedath/slm-125m-base on a grounded Q&A instruction dataset covering legal, financial, and general text.

Training data

7,521 chat-format (system/user/assistant) examples generated from a legal/SEC/web corpus using Gemini (gemini-3.1-flash-lite) as teacher, filtered by an LLM-as-judge pass for grounding/faithfulness, deduplicated by embedding similarity, covering four task types: grounded QA, summarization, extraction, and rewriting.

Training

Full fine-tune (not LoRA), 3 epochs, 1x H100, dynamic per-batch padding, loss masked to the assistant response only. 354 steps, ~1.6M real tokens processed, 69 seconds total. Validation perplexity 9.13 on the Q&A task.

Usage

Prompts use the same special-token format as the base model:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("sahuabinash/slm-125m-finetuned-qa")
model = AutoModelForCausalLM.from_pretrained("sahuabinash/slm-125m-finetuned-qa")

prompt = "<|bos|><|system|>Answer the question using only the given passage.<|user|>What is a plaintiff?<|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80)
print(tok.decode(out[0], skip_special_tokens=True))