Downloads · 30 days
5
12% of all-time downloads
slm-125m-base
ankitw497/slm-125m-base
slm-125m-base is a machine learning model from ankitw497. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
A 125.8M-parameter LLaMA-architecture language model trained from scratch on a legal/financial/web data mix.
All-time downloads
43
Public
Parameters
126M
503 MB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors503 MB · 100%
At a glance
- License
- mit
- Model type
- llama
- Access
- Public
- Created
- Jul 10, 2026
- Updated
- Jul 10, 2026
- SHA
- c3edb8d8
Model card
From the Hugging Face model README
ankitw497/slm-125m-base
A 125.8M-parameter LLaMA-architecture language model trained from scratch on a legal/financial/web data mix.
Model
- 12 layers, 768 hidden, 12 heads (MHA), vocab 16,384, context 1024, tied embeddings
- Trained with a custom 16K byte-level BPE tokenizer (fit on the training corpus)
Data
Legal-first mix (NOT 70/20/10): case-law ~40%, SEC filings ~40%, fineweb-edu ~20%. Cleaned, deduplicated (exact + MinHash near-dup), and decontaminated against CaseHOLD/LexGLUE eval sets. See the source repo's REPLICATION_GUIDE.md for the full pipeline.
Training
- 2.04B train tokens, 1 epoch, seq_len 1024
- 1x H100, AdamW, cosine LR schedule with warmup
- Final val loss: 2.3263 (perplexity 10.24)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("ankitw497/slm-125m-base")
model = AutoModelForCausalLM.from_pretrained("ankitw497/slm-125m-base")
- Type
- llama
- License
- mit
- Created
- Jul 10, 2026
- Updated
- Jul 10, 2026