slm-125m-base

SLM-125M · a legal & financial base model

A 125-million-parameter, Llama-style decoder for a legal-and-financial-first corpus: US case law, SEC filings, and educational web text. It is a compact base model, fluent in the register of law and finance.

This is a research artifact, not an instruction-tuned assistant. It continues text; it does not answer questions or follow instructions, and it has no safety alignment. Do not rely on it for legal, financial, or other professional advice.

Highlights

Architecture

Standard Llama-family decoder, mapped 1:1 to transformers.LlamaConfig:

Training data

The two legal sources are small by nature, so the mix is legal-first (~40 / 40 / 20) rather than the usual web-heavy blend. Realized token counts over the continuation corpus:

Data pipeline: stream three public datasets → 6-step deterministic cleaning (line filtering, boilerplate stripping, repetition and language checks, an OCR-garble gate on scanned opinions) → dedup + decontamination (MinHash-LSH near-duplicate removal + 13-gram eval-set stripping) → a fresh 16,384 byte-level BPE tokenizer → pack into 1,024-token windows with a 99/1 train-val split. The full pipeline runs on CPU for under $1.

Training procedure

<!-- This release **continues pretraining** from [`thesreedath/slm-125m-base`](https://huggingface.co/thesreedath/slm-125m-base) (a from-scratch 125M model, reportedly 5 epochs) for **one additional epoch**. -->

The full data pipeline and the 16K tokenizer here were built from scratch; the weights are warm-started from another base checkpoint.

<!-- 🔎 **Live playground:** https://slm-125m-base.vercel.app -->

This release continues pretraining from another release where the base checkpoint was reportedly pretrained for 5 epochs. This release continues it for 1 additional epoch via warm-start: weights loaded from the base checkpoint with a fresh AdamW optimizer and a gentle cosine schedule (peak LR 1e-4 → 6e-5, 20M-token warmup) on 8×H100. Validation perplexity improved to 8.41. Special-token ids: <|bos|>=0, <|eos|>=1, <|pad|>=2.

The continuation epoch used our own re-cleaned corpus - the same source distribution as the base's data, but not byte-identical - so it is best described as "5 epochs on the base data plus 1 on ours," not a literal 6th pass over the identical token stream.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("akhil-2019/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("akhil-2019/slm-125m-base")

prompt = "The plaintiff alleges that the defendant"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, do_sample=True,
                     temperature=0.8, top_p=0.95, top_k=50,
                     repetition_penalty=1.3, no_repeat_ngram_size=3,
                     eos_token_id=1, pad_token_id=2)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Limitations & intended use

• Base completer only - no instruction-following, chat, or safety tuning.
• Can produce fluent-sounding but incorrect or outdated statements; knowledge is bounded by the training data.
• English-only; specialized to legal/financial prose and weaker on general text.
• Not for legal, financial, or other professional advice. Outputs are text continuations, not verified facts.

License

Released under Apache-2.0. Underlying datasets retain their own licenses.