slm-125m-base

slm-125m-base

A 126M-parameter Llama-architecture base model pretrained from scratch on US case law, SEC filings, and educational web text. Trained on 8xH100 for ~28 minutes at a total project cost of roughly $18.

This is a base model. It has had no instruction tuning, no RLHF, and no safety alignment. It completes text; it does not follow instructions.

Architecture

Training data

1.92B tokens, ~76% legal:

Pipeline: deterministic rule-based cleaning (line filters, boilerplate stripping, repetition and language gates, plus a dictionary-based OCR gate on case law) then MinHash near-dedup, exact dedup, and 13-gram decontamination against the CaseHOLD evaluation set.

Training

2 epochs (3.84B tokens seen), AdamW, cosine schedule to a 6e-05 cooldown, bf16, sequence length 1024, global batch 524,288 tokens.

Results

CaseHOLD is measured on data excluded from training by 13-gram decontamination, so it is not inflated by leakage. Expect near-chance zero-shot performance: a 125M base model with no task tuning is not competitive on multiple-choice legal reasoning, and this number is a baseline for fine-tuning, not a capability claim.

Limitations

Small, base-only, English-only, 1024-token context. Case-law inputs contain OCR noise that survives filtering. Not legal or financial advice and not suitable for any use where an incorrect statement of law carries consequences.