slm-125m-base

slm-125m-base

A 125.8M-parameter Llama-style causal LM pretrained from random weights on a legal-first corpus. No distillation, no fine-tuning from an existing checkpoint: the tokenizer, the corpus and the weights were all built from scratch.

Architecture

Because embeddings are tied, model.safetensors contains no lm_head.weight — it is shared with the input embedding. That is expected, not a truncated upload.

Training data

Legal-first mix, cleaned and deduplicated from scratch (~40/40/20):

Pipeline: quality/language/OCR filtering, MinHash near-dedup, exact dedup, and 13-gram decontamination against the CaseHOLD benchmark via LexGLUE's case_hold config, which is held out. That removed 24,002 case-law and 175 SEC documents. Result: 2.04B training tokens packed into 1024-token windows.

Contamination caveat (measured, please read before benchmarking)

Decontamination used LexGLUE's case_hold config (480,908 13-grams). A later audit found the standalone casehold/casehold dataset contributes 796,649 additional 13-grams that were never screened — LexGLUE covers only 38% of them. 5.1% of the retained corpus matches at least one (case-law 5.69%, SEC 2.35%).

Treat that 5.1% as an upper bound rather than a leakage rate: SEC filings should contain no CaseHOLD case-law text, so their 2.35% is largely innocent collision on legal boilerplate, and a 13-gram overlap in a citing context is not the same as leaking CaseHOLD's multiple-choice answer. Still, do not quote a CaseHOLD score for this model and treat any LexGLUE case_hold number as optimistic. The honest metric for this model is held-out perplexity on its own validation split, which is what is reported above.

Training

Held-out validation perplexity: 10.42.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("narendraalluri/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("narendraalluri/slm-125m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=80, min_new_tokens=40, do_sample=True,
                     temperature=0.8, top_p=0.95, top_k=50, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

Pass min_new_tokens — a base model at this scale will happily emit EOS immediately.

Honest limitations

This is a base completer, not an assistant. It has never seen instruction data or a chat template (the tokenizer carries <|user|>/<|assistant|> tokens, but training never used them), so it will not follow instructions or answer questions.

It writes fluent legal and financial prose with correct register and citation form, and it invents the content: case names, reporter numbers and dollar figures are plausible fabrications. Perplexity is the honest metric here; factual grounding needs retrieval. Do not use it for legal advice.