slm125m-legal-base

slm125m-legal-base

A 125.8M-parameter Llama-architecture language model, pretrained from random weights on a purpose-built 2.04B-token legal and financial corpus. No fine-tuning, no distillation — the data pipeline, the tokenizer, and the model were all built from scratch.

Total cost to build: ~$31. Validation perplexity: 9.20.

---

⚠️ Read this first

This model invents legal citations in perfect format. It learned the shape of legal writing, not its content.

From an actual sample output:

"...the exclusionary rule of the Fourth Amendment 'does not require that the evidence be excluded at all.' Coolidge v. New Hampshire, 403 U.S. 443, 454 (1971)"

Coolidge v. New Hampshire is a real case. The citation format is correct. The quote is fabricated. A lawyer skimming that would not immediately notice.

It is also bad at arithmetic. In a financial sample it wrote "increased to $487.2 million from $393.9 million, an increase of $125.0 million or 55%" — the correct figures are ~$93.3M and ~23.7%. Both numbers are simply wrong.

This is a base completer. It knows form, not fact.

• ❌ Not legal advice. Do not rely on any factual claim it makes.
• ❌ Not a chat model. It continues text; it does not answer questions.
• ❌ Not a knowledge base. At 125M parameters it cannot store reliable facts.
• ✅ Good for: studying legal register, text completion research, as a RAG generator grounded in retrieved documents, or as a base for fine-tuning.

If you need factual legal output, you need retrieval. No amount of pretraining at this scale fixes hallucinated citations.

---

Try it

👉 Run it in Colab — free CPU, no GPU needed, loads in seconds.

Or locally — the model is ~500MB (fp32) and runs on CPU:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "karnan/slm125m-legal-base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "In determining whether the search was reasonable, the court"
ids = tok(prompt, return_tensors="pt")["input_ids"]   # note: input_ids only

out = model.generate(
    input_ids=ids,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
)
print(tok.decode(out[0], skip_special_tokens=True))

Note: pass input_ids explicitly. This tokenizer also emits token_type_ids, which LlamaForCausalLM.generate() rejects. **tok(prompt) will raise a ValueError.

Give it a prefix, not a question. "What is a tort?" will get you a rambling continuation; "A tort is" will get you a definition.

What it actually produces

Fourth Amendment analysis — correct doctrinal framing, fabricated quote:

In determining whether the search was reasonable, the court must assess its effect on the intrusion on an individual's privacy. The United States Supreme Court has stated that the exclusionary rule of the Fourth Amendment "does not require that the evidence be excluded at all." Coolidge v. New Hampshire, 403 U.S. 443, 454 (1971)...

Enumerated civil pleading — genuinely well-formed legal reasoning:

The plaintiff alleges that the defendant 's failure to comply with this provision of the contract made the building of the plaintiff's property and premises impossible, because (1) the defendant was without authority to enter into a lease for any purpose, (2) the plaintiff had no adequate remedy at law; (3) the defendant breached the implied covenants in the lease by failing to maintain the common...

Contract / default clause — correct structure and register:

Pursuant to the terms of this Agreement, upon the occurrence of an Event of Default, General Partner shall have the option to: (i) pay or commence a voluntary case under Chapter 11 of the Bankruptcy Code; (ii) make all payments required to be made in accordance with the payment terms of the Notes...

SEC-style MD&A — fluent prose, wrong arithmetic:

The Company's net revenues for the fiscal year ended December 31, 1996 increased to $487.2 million from $393.9 million, an increase of $125.0 million or 55% over the comparable period of 1995...

Note it switches register correctly between legal and financial prompts — the domain training worked. The facts did not.

Model

The tokenizer is domain-trained, not borrowed. "Petition for a writ of certiorari to the United States Court of Appeals is denied" encodes to 17 tokens — a general-purpose tokenizer spends several on certiorari alone. Byte-level means no UNK tokens ever.

Training data

2.04B tokens, built from three public datasets:

The mix is legal-first, not 70/20/10. The legal sources are supply-bound — measured, they cap at ~2B tokens combined — so a 70% case-law corpus at 10B tokens is arithmetically impossible. We take all of both legal sources and cap the web filler at 0.5B. The web slice is not padding: a pure-legal corpus produces a model that has never seen ordinary English.

Pipeline: 6-step deterministic cleaning → MinHash-LSH near-dedup → exact dedup → decontamination against CaseHOLD/LexGLUE (23,995 documents removed) → 16K BPE → packed uint16 1024-token windows, 99/1 split.

The decontamination matters: without it, the perplexity below would be measuring memorization of the eval set rather than learning.

Training

5 epochs (10.20B tokens seen), 2×H100, 19,448 steps, ~4 hours, $29.22. AdamW (β 0.9/0.95, wd 0.1), cosine 6e-4 → 6e-5, 200M-token warmup, 0.5M-token batches.

Perplexity is the only metric reported. At 125M parameters, MMLU-style benchmarks are near-random noise; publishing them would be theatre.

Cost

A prior implementation of this same project cost $56.58 for the identical 5 epochs. The ~2× reduction came from torch.compile + flash attention (SDPA) + fused AdamW + TF32 — measured, by benchmarking both an H100 (420K tok/s) and an A100-80GB (173K tok/s) before spending anything. The H100 turned out to be cheaper per token despite costing 1.58× more per hour, because it is 2.4× faster.

Two bugs worth knowing about

1. The OCR filter was deleting 58% of the corpus. The obvious threshold — drop a document if >20% of its words are absent from /usr/share/dict/words — is catastrophic for legal text. That dictionary is a spelling dictionary: no inflections (alleges), no legal register (hereto), no Latin (certiorari), no citations. Clean legal prose scores a median 0.21 against it. The 0.20 cutoff sat directly on the mode of the distribution. Real OCR garbage scores ~0.96 — the populations are trivially separable, and 0.20 was simply on the wrong side of the wrong one. Corrected to 0.35.

2. The checkpoint-resume path was broken. torch.save/load round-trips the RNG state to a non-ByteTensor, so torch.set_rng_state raised on every resume. Modal preempts every GPU function by default — there is no non-preemptible tier at any price — so every preemption would have killed the run. Caught by a deliberate $0.20 kill/resume drill before committing to a 4-hour unattended run.

An honest negative result

The prior implementation reached val ppl 8.50. This model reached 9.20 — we lost by 0.70.

I had predicted the opposite. I attributed the prior run's late-curve flatline (8.62 → 8.50 over its final 6,000 steps) to its use of bf16 master weights, which round small updates to zero once the learning rate decays. That mechanism is real — but I inferred causation from a curve shape and never tested the null hypothesis, which is simply that a converged cosine schedule flattens at its tail.

The likely actual cause of the gap: this corpus is 7% smaller (2.04B vs 2.19B tokens), because the domain tokenizer compresses legal text better. At the same 5 epochs, this run saw 10.20B tokens against the prior run's 10.95B. Fewer tokens seen → worse perplexity.

The irony: the tokenizer working better is part of why the headline number looks worse, because holding epochs constant does not hold tokens seen constant.

Reproduce it

Every phase is one worker per data shard (Modal preempts by default, so a preemption costs one shard, not the run):

modal run modal_app.py::clean       # stream + clean       ~15 min
modal run modal_app.py::dedup       # dedup + decontam     ~6 min
modal run modal_app.py::tokenizer   # 16K BPE              ~4 min
modal run modal_app.py::tokenize    # pack uint16 windows  ~10 min
modal run modal_app.py::drill       # GATE: prove resume works
modal run modal_app.py::benchmark   # measure $/1B-tokens per GPU
modal run modal_app.py::pretrain --gpu H100 --max-usd 40

License

Apache 2.0. Training data is public: HFforLegal/case-law, PleIAs/SEC, HuggingFaceFW/fineweb-edu — consult each for its own terms.

Not legal advice. Not a source of legal facts.