Downloads · 30 days
11
18% of all-time downloads
slm-125m-extended
Ace-2504/slm-125m-extended
slm-125m-extended is a text generation model from Ace-2504. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
Ace-2504/slm-125m-base continued for one epoch on ~515M additional tokens (389M FineWeb-Edu + 131M US case law) that the base model had never seen.
All-time downloads
60
Public
Parameters
126M
503 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors503 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Model type
- llama
- Access
- Public
- Created
- Jul 20, 2026
- Updated
- Jul 20, 2026
- SHA
- a4841640
Try a prompt
Model card
From the Hugging Face model README
SLM-125M-extended — continued pretraining of slm-125m-base
Ace-2504/slm-125m-base continued for one epoch on ~515M additional tokens
(389M FineWeb-Edu + 131M US case law) that the base model had never seen.
Constant learning rate 1e-4, optimizer state reset, 982 steps on one A100-40GB.
This model exists to answer a research question: does continued pretraining on new data improve a small model overall, or does it cause catastrophic forgetting?
Measured against the base model on its own held-out data, forgetting tracked each domain's share of the new data almost exactly:
| Domain | share of new data | loss change |
|---|---|---|
| SEC filings | 0% | +8.1% (worst) |
| US case law | 25% | +1.0% |
| FineWeb-Edu | 75% | -1.7% (improved) |
Use Ace-2504/slm-125m-base if you want the original. Neither model is
instruction-tuned; both are base models.
- Task
- Text Generation
- Library
- transformers
- Type
- llama
- License
- mit
- Created
- Jul 20, 2026
- Updated
- Jul 20, 2026