slm125mLIVE-base

slm125mLIVE-base

A 125.8M-parameter Llama-architecture language model pretrained from scratch on a legal and financial corpus. Base model, no instruction tuning.

Corpus

2.50B unique tokens. Cleaned, MinHash-LSH deduplicated, and decontaminated against CaseHOLD / LexGLUE case_hold by 13-gram overlap - 24,002 case-law documents (10.3% of that source) removed.

Architecture

<|user|>, <|assistant|> and <|system|> are reserved in the vocabulary, so post-training needs no embedding resize.

Training

WSD schedule (warmup - stable - decay), AdamW, bf16 on 4xH100.

The earlier checkpoint, and why it is worth knowing about

An earlier revision of this repo held a checkpoint at perplexity 11.06. Further pretraining brought it to 9.71 - a 12% improvement, for about $11 of GPU time.

Judged answer quality moved from 0.38 to 0.47 out of 10, a change whose confidence interval includes zero. The perplexity gain was real and the quality gain was not measurable. Post-training moved the same metric by more than 2.5 points for under a dollar.

Both checkpoints are reachable: this revision is the 9.71 model, and commit 560c780c55 is the 11.06 one, so the comparison can be reproduced rather than taken on trust.

Leaderboard

147 held-out questions, decontaminated against the SFT training passages, scored 0-10 by an LLM judge. Every model was scored on the same item by the same judge with model identities hidden and re-shuffled per item, so a gap between two rows is a gap between two models rather than between two judging runs. Intervals are 10,000-sample bootstraps.

Paired step-over-step change on the same items:

Intended use

Research and further post-training. At 125M parameters this is a base for building pipelines, not an assistant. Expect locally fluent, frequently incoherent generation - it scores 0 on 63% of held-out questions. Not legal or financial advice.