Downloads · 30 days
12
34% of all-time downloads
winnow-olmoe-math-keep50
hbfreed/winnow-olmoe-math-keep50
winnow-olmoe-math-keep50 is a machine learning model from hbfreed. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Channel-level pruned + healed OLMoE-1B-7B-0125-Instruct. Requires trustremotecode=True (ragged variable-width experts).
All-time downloads
35
Public
Parameters
3.7B
7.4 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors7.4 GB · 100%
At a glance
- License
- apache-2.0
- Model type
- pruned_olmoe
- Access
- Public
- Created
- Aug 9, 2026
- Updated
- Aug 21, 2026
- SHA
- da942d3e
Base models
Model card
From the Hugging Face model README
winnow-olmoe-math-keep50
Channel-level pruned + healed OLMoE-1B-7B-0125-Instruct.
Requires trust_remote_code=True (ragged variable-width experts).
| Base | allenai/OLMoE-1B-7B-0125-Instruct |
| Params | 3.70B |
| Experts fully deleted | 217/1024 |
| Keep fraction | 0.50 |
| Criterion | channel-level REAP, per-layer budgets, block 128, min width 128 |
| Calibration | Dolmino-math (scores_0125inst_dolmino-math) |
| Final forward top-128 KL | 0.031 |
Healing
Off-policy forward-KL distillation against cached top-128 teacher targets
(dolci_math_curated_opd_top128), 150 steps, 120k loss tokens/step (18M total), AdamW8bit,
lr 3e-5, wd 0.1, 10 warmup steps, grad clip 1.0, chat frames, max seq len 2048, seed 1223.
All six models in this release share this recipe and are matched on optimizer steps and loss tokens. Note they are not matched on data exposure: the math cache is 6.48M unique tokens (~2.8 epochs at 150 steps) while the general cache is 22.3M (~0.81 epochs).
- Type
- pruned_olmoe
- License
- apache-2.0
- Created
- Aug 9, 2026
- Updated
- Aug 21, 2026