Downloads · 30 days
10
50% of all-time downloads
WattsIshaan/OLMo-20m-64B-adamw-cosine-10percent
OLMo-20m-64B-adamw-cosine-10percent is a machine learning model from WattsIshaan. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-4.0.
OLMo-20M checkpoint pretrained with AdamW optimizer and cosine LR schedule on 64B tokens from DCLM. This checkpoint is for 90% of the training steps (220000 steps at 256 bsz), and can be annealed for the remaining 10%…
Downloads · 30 days
10
50% of all-time downloads
All-time downloads
20
Public
Repo size
717 MB
Likes
0
Public
Click a slice to open those files.
.pt717 MB · 100%
From the Hugging Face model README
OLMo-20M checkpoint pretrained with AdamW optimizer and cosine LR schedule on 64B tokens from DCLM. This checkpoint is for 90% of the training steps (220000 steps at 256 bsz), and can be annealed for the remaining 10% steps.
Paper:"Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting" Ishaan Watts*, Catherine Li*, Sachin Goyal, Jacob Mitchell Springer, Aditi Raghunathan ICML2026 https://arxiv.org/abs/2605.02105