slm125m-live-base

slm-125m-base

A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal/financial corpus. No weights or tokenizer were inherited from any existing model.

Model

Training data (2.04B tokens)

Streamed from HuggingFace, cleaned, deduplicated and decontaminated:

Realized mix is roughly 40/40/20 (case-law / SEC / web), i.e. ~78% legal. The legal sources together contain only ~2B clean tokens, so a legal-heavy mix at a larger scale is not achievable with these corpora.

Pipeline: 6-step deterministic cleaning (line filters, boilerplate strip, repetition, language ID, OCR-garble gate on scanned case law) -> MinHash LSH near-dedup + exact dedup -> 13-gram decontamination against the CaseHOLD / LexGLUE case_hold evaluation set -> 16K byte-level BPE -> packed uint16 1024-token windows, 99/1 train/val split.

Training

• 2 epochs over the fixed corpus (~4.08B tokens seen)

• 8x H100, DDP, bf16 autocast, torch.compile, fused AdamW, SDPA attention

• global batch 524,288 tokens, cosine LR 0.0006 -> 6e-05, 200M warmup tokens

• measured throughput ~3.19M tok/s (35% MFU)

• final val loss: 2.2083

• final val perplexity: 9.10

Intended use and limitations

A small base model for research on legal/financial text. It is not instruction-tuned, has no safety alignment, and will produce fluent but unreliable legal or financial statements. Do not use it for legal or financial advice. Case-law inputs include OCR noise; SEC filings skew to older filings. The corpus is decontaminated against CaseHOLD, so CaseHOLD evaluation is meaningful; no other benchmark was held out.