slm-125m-base

slm-125m-base

A 125.8M-parameter LLaMA-architecture causal language model, pretrained from scratch on a legal/financial/general-web corpus.

• Architecture: LLaMA (12 layers, 768 hidden, 12 heads, tied embeddings), 16,384-vocab byte-level BPE tokenizer, 1024 context length.
• Training data: ~2.04B tokens — US case law (HFforLegal/case-law), SEC filings (PleIAs/SEC), and general web text (HuggingFaceFW/fineweb-edu), cleaned, deduplicated, and decontaminated against LexGLUE/CaseHOLD.
• Training: 1 epoch, full cosine LR schedule (6e-4 → 6e-5), AdamW, on 8xH100.
• Validation perplexity: 11.04 (val loss 2.40) on a held-out 1% split.

This is a base (not instruction-tuned) model — it continues text, it does not follow chat instructions.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("sahuabinash/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("sahuabinash/slm-125m-base")

inputs = tok("The plaintiff shall", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0]))