Downloads · 30 days
46
7% of all-time downloads
slm-125m-base
ppanja/slm-125m-base
slm-125m-base is a text generation model from ppanja. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
125M-parameter causal language model trained from scratch on a legal/financial corpus.
All-time downloads
617
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors503 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Jul 13, 2026
- Updated
- Jul 13, 2026
- SHA
- 8fb136d5
Try a prompt
Model card
From the Hugging Face model README
slm-125m
125M-parameter causal language model trained from scratch on a legal/financial corpus.
Model
| Architecture | Llama-style (SwiGLU, RoPE, RMSNorm) |
| Parameters | ~125.8M |
| Layers / dim / heads | 12 / 768 / 12 |
| Context | 1024 tokens |
| Vocab | 16384 (byte-level BPE) |
Training data
| Source | Role |
|---|---|
| HFforLegal/case-law | US case law |
| PleIAs/SEC | SEC filings |
| HuggingFaceFW/fineweb-edu | General educational web text |
Packed train tokens: unknown | val tokens: unknown
Eval benchmarks (LexGLUE, CaseHOLD) were held out during corpus construction.
Training run
| Hyperparameter | Value |
|---|---|
| Global batch (tokens) | 524,288 |
| LR / min LR | 0.0006 / 6e-05 |
| Warmup tokens | 200M |
| Weight decay | 0.1 |
Last logged train loss: n/a | val loss: n/a
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ppanja/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("ppanja/slm-125m-base")
inputs = tok("The plaintiff shall bear the burden of proof", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=64)
print(tok.decode(out[0]))
Chat special tokens (<|user|>, <|assistant|>, <|system|>) are in the vocabulary for future instruction tuning.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Created
- Jul 13, 2026
- Updated
- Jul 13, 2026