fine-tuned-125m-slm

SLM-125M — SFT data-scaling study

Supervised fine-tunes of Ace-2504/slm-125m-base (125.8M params, 1024 ctx, pretrained from scratch on 2.04B tokens of US case law, SEC filings and educational web text).

This repo is a research artifact, not a useful assistant. It is the output of a ten-round data-scaling study whose headline result is a negative one: scaling SFT data 10× did not improve answer quality (judge flat at ~1.5/5) and tripled catastrophic forgetting.

📊 Full write-up: https://ace-2504.github.io/fine-tuned-125m-slm/ · 💻 Code: https://github.com/Ace-2504/fine-tuned-125m-slm

Two revisions — pick deliberately

The two judge scores are identical within the measured ±0.07 noise band, so main (day 2) strictly dominates: same quality, half the damage, one-fifth the data.

from transformers import AutoModelForCausalLM, AutoTokenizer

# the better model (day 2)
tok = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm")
model = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm")

# the 10k endpoint, for comparison
tok10 = AutoTokenizer.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")
model10 = AutoModelForCausalLM.from_pretrained("Ace-2504/fine-tuned-125m-slm", revision="day-10")

msgs = [
    {"role": "system", "content": "You are a precise legal and financial assistant."},
    {"role": "user", "content": "What is the standard of proof in a civil lawsuit?"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt")
print(tok.decode(model.generate(ids, max_new_tokens=120)[0][ids.shape[1]:], skip_special_tokens=True))

Honest limitations

• Answer quality is poor (~1.5/5). It learned the shape of an assistant turn, not reliable answers. Expect repetition, prompt echoing, and confident wrong answers.
• It forgot part of its pretraining. Perplexity on the original pretraining validation set rose from 11.35 → 12.42 (main) / 13.20 (day-10).
• Root cause: ~40% of training was closed-book QA — recalling a document fact from a single exposure, which a 125M model cannot do. Common facts land; long-tail specifics never do.
• Do not use for legal or financial advice. It is a study artifact.

Training

Full fine-tune (no LoRA), 3 epochs, batch 16, LR 3e-5→3e-6 cosine, AdamW, bf16, assistant-only loss masking, seed 1337, on a single Modal L4. Data: QnA written by gemini-3.1-flash-lite, grounded in passages from the same corpus the base was pretrained on, then filtered, deduplicated (exact + embedding), decontaminated and balanced. Chat template uses <|system|> / <|user|> / <|assistant|>.

generation_config ships with repetition_penalty=1.3 and no_repeat_ngram_size=3 — without them greedy decoding falls into repetition loops.