Downloads · 30 days
13
42% of all-time downloads
winnow-olmoe-math-keep75
hbfreed/winnow-olmoe-math-keep75
winnow-olmoe-math-keep75 is a machine learning model from hbfreed. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Channel-level pruned + healed OLMoE-1B-7B-0125-Instruct. Requires trustremotecode=True (ragged variable-width experts).
All-time downloads
31
Public
Parameters
5.3B
10.6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors10.6 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- pruned_olmoe
- Access
- Public
- Created
- Aug 9, 2026
- Updated
- Aug 21, 2026
- SHA
- 232d69dd
Base models
Model card
From the Hugging Face model README
winnow-olmoe-math-keep75
Channel-level pruned + healed OLMoE-1B-7B-0125-Instruct.
Requires trust_remote_code=True (ragged variable-width experts).
| Base | allenai/OLMoE-1B-7B-0125-Instruct |
| Params | 5.31B |
| Experts fully deleted | 88/1024 |
| Keep fraction | 0.75 |
| Criterion | channel-level REAP, per-layer budgets, block 128, min width 128 |
| Calibration | Dolmino-math (scores_0125inst_dolmino-math) |
| Final forward top-128 KL | 0.013 |
Healing
Off-policy forward-KL distillation against cached top-128 teacher targets
(dolci_math_curated_opd_top128), 150 steps, 120k loss tokens/step (18M total), AdamW8bit,
lr 3e-5, wd 0.1, 10 warmup steps, grad clip 1.0, chat frames, max seq len 2048, seed 1223.
All six models in this release share this recipe and are matched on optimizer steps and loss tokens. Note they are not matched on data exposure: the math cache is 6.48M unique tokens (~2.8 epochs at 150 steps) while the general cache is 22.3M (~0.81 epochs).
- Type
- pruned_olmoe
- License
- apache-2.0
- Created
- Aug 9, 2026
- Updated
- Aug 21, 2026