Downloads · 30 days
0
lm-100m-en-overtrained
Aeryx-ai/lm-100m-en-overtrained
lm-100m-en-overtrained is a machine learning model from Aeryx-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
The ~26B-token overtrained base of the A/B below. trunk/ is the pure WSD stable trunk (step 200k); annealed/ is the cosine-decayed final base used for SFT.
Access
Public
Updated Jun 18, 2026
Repo size
1.2 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt1.2 GB · 100%
At a glance
- Access
- Public
- Created
- Jun 18, 2026
- Updated
- Jun 18, 2026
- SHA
- 9a2588f7
Model card
From the Hugging Face model README
100M English LM (overtrained base)
The ~26B-token overtrained base of the A/B below. trunk/ is the pure WSD stable trunk (step 200k); annealed/ is the cosine-decayed final base used for SFT.
Experiment: does overtraining a tiny LM pay off?
Two ~100M-parameter from-scratch English language models, identical architecture and identical instruct SFT, differing only in pretraining token budget:
| model | pretrain tokens | instruct val loss |
|---|---|---|
| overtrained | ~26B (~260 tok/param) | 1.7337 |
| chinchilla (compute-optimal floor) | ~2B (~20 tok/param) | 2.0180 |
After identical SFT, the heavily overtrained base yields a measurably more instruction-shaped model: ~0.28 nats lower validation loss. Trained end to end on a single consumer RTX 3070 Ti (8GB).
Architecture & data
~100M-parameter vanilla Llama (RMSNorm, SwiGLU, RoPE, grouped-query attention, tied embeddings), 32,000-token ByteLevel BPE, 2048 context. English-only corpus (cosmopedia-v2 + fineweb-edu-dedup + wikipedia-en), trained from scratch.
Custom architecture: load with the included model.py + config.py, not the HF
AutoModel API. Instruct models use ChatML with assistant-only loss masking.
- Created
- Jun 18, 2026
- Updated
- Jun 18, 2026