slm-125m-base

slm-125m-base

A 125M-parameter Llama-style base language model pretrained from scratch — tokenizer, data pipeline, and training loop included — on a legal/financial corpus (US case law + SEC filings + a web slice, ~2.19B tokens).

🎮 Try it in your browser (WebGPU, no install): slm-125m-playground.vercel.app — the int8 ONNX export runs fully client-side via 🤗 Transformers.js (~139 MB one-time download, then cached/offline).

This is a base completer, not a chat model. Prompt it with the start of a sentence and it continues fluently in the legal/financial register. It does not answer questions, follow instructions, or reliably state facts — at this size, world knowledge is capped at roughly ~2 bits/param. Instruction-tuned (SFT) and preference-tuned variants are on the roadmap below.

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("genaiquest/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("genaiquest/slm-125m-base")

ids = tok("The plaintiff alleges that the defendant", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=60, do_sample=True,
      temperature=0.8, min_new_tokens=40, repetition_penalty=1.3)[0]))

Suggested sampling: temperature 0.8 · top_p 0.95 · repetition_penalty 1.3 (small base models loop without a repetition penalty).

Prompts that work well (in-register continuations):

• IN THE UNITED STATES DISTRICT COURT
• Pursuant to Section 10(b) of the Securities
• The Company's revenue for the fiscal year
• The court finds that the evidence presented

Architecture

Deliberately vanilla geometry, mapping 1:1 to transformers.LlamaConfig — so it exports to ONNX (and GGUF) with zero custom-converter work. That design choice is exactly what makes the in-browser playground possible.

Parameter budget (tied):

embed      16,384 × 768        = 12.58M
attn/layer 4 × 768²            =  2.36M
mlp/layer  3 × 768 × 3072      =  7.08M
(attn + mlp) × 12 layers       = 113.2M
total                          ≈ 125.8M

Training

• Data: US case law + SEC filings + a general web slice, deduplicated; ~2.19B tokens total. Case law and SEC filings are public-domain US government works. [40%-40%-20%]
• Tokenizer: 16,384-token BPE trained from scratch on the corpus.
• Compute: 1 epoch on 8×H100. [FILL: wall-clock hours, tokens/sec, approx cost — e.g., "X hrs ≈ $Y at market spot rates"]
• Recipe: optimizer (AdamW), peak LR, schedule (cosine), 200M token warmup, global batch size 32, 0.5M tokens/step, grad clip, weight decay
• Scaling note: 2.19B tokens ÷ 125.8M params ≈ 17.5 tokens/param, close to the Chinchilla compute-optimal ratio (~20) — this run prioritized compute-optimality over deliberate overtraining.

Loss curve

Evaluation

Perplexity is measured on a held-out slice of the training distribution (legal/financial register). Expect materially higher perplexity on out-of-domain text; this model is a domain specialist by construction.

Intended use & limitations

Intended: studying small-model pretraining end to end; domain-register text continuation; a base checkpoint for SFT/DPO/RAG experiments; edge/browser inference demos; teaching material for the full train→quantize→deploy pipeline.

Not intended: legal or financial advice, factual question answering, chat, or any production decision-making. Outputs can be fluent and wrong — at 125M parameters that is a certainty, not a risk. Generated text may echo the style of court filings and SEC documents; verify nothing here as fact.

ONNX / browser deployment

The repo includes an int8 (q8) ONNX export consumed by 🤗 Transformers.js in the playground. Because the architecture maps 1:1 to LlamaConfig, the export pipeline is: bf16 safetensors → ONNX → dynamic int8 quantization, with no custom conversion code.

Roadmap

This is checkpoint 1 of a series building the full enterprise-SLM stack in public:

• ✅ slm-125m-base — pretraining from scratch (this model)
• ⏳ -sft — supervised fine-tuning for instruction following / QnA
• ⏳ -dpo/ppo — preference tuning
• ⏳ RAG-grounded variants for document QnA with citations
• ⏳ Serving optimization study on this model family (vLLM, CUDA graphs, FP8) — companion to my vLLM H100 serving experiments

Author

Bhabani Nayak — AI engineering leader; agentic AI at consumer scale (Sephora; previously Shopify, Twitter, Apple). IEEE Senior Member. LinkedIn · Substack · Playground

Citation

@misc{nayak2026slm125m,
  author = {Nayak, Bhabani},
  title  = {slm-125m-base: a 125M Llama-style language model pretrained from scratch on legal/financial text},
  year   = {2026},
  url    = {https://huggingface.co/genaiquest/slm-125m-base}
}