Downloads · 30 days
0
lm-100m-en-overtrained-instruct
Aeryx-ai/lm-100m-en-overtrained-instruct
lm-100m-en-overtrained-instruct is a machine learning model from Aeryx-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Instruct SFT of the overtrained base. Validation loss 1.7337 — the winning arm of the A/B below.
Access
Public
Updated Jun 18, 2026
Repo size
601 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt601 MB · 100%
At a glance
- Access
- Public
- Created
- Jun 18, 2026
- Updated
- Jun 18, 2026
- SHA
- 39bf67c6
Model card
From the Hugging Face model README
100M English LM (overtrained + instruct SFT)
Instruct SFT of the overtrained base. Validation loss 1.7337 — the winning arm of the A/B below.
Experiment: does overtraining a tiny LM pay off?
Two ~100M-parameter from-scratch English language models, identical architecture and identical instruct SFT, differing only in pretraining token budget:
| model | pretrain tokens | instruct val loss |
|---|---|---|
| overtrained | ~26B (~260 tok/param) | 1.7337 |
| chinchilla (compute-optimal floor) | ~2B (~20 tok/param) | 2.0180 |
After identical SFT, the heavily overtrained base yields a measurably more instruction-shaped model: ~0.28 nats lower validation loss. Trained end to end on a single consumer RTX 3070 Ti (8GB).
Architecture & data
~100M-parameter vanilla Llama (RMSNorm, SwiGLU, RoPE, grouped-query attention, tied embeddings), 32,000-token ByteLevel BPE, 2048 context. English-only corpus (cosmopedia-v2 + fineweb-edu-dedup + wikipedia-en), trained from scratch.
Custom architecture: load with the included model.py + config.py, not the HF
AutoModel API. Instruct models use ChatML with assistant-only loss masking.
- Created
- Jun 18, 2026
- Updated
- Jun 18, 2026