Downloads · 30 days
0
phi-mini-moe-pruned-specialized-v1
maxie-12321/phi-mini-moe-pruned-specialized-v1
phi-mini-moe-pruned-specialized-v1 is a machine learning model from maxie-12321. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This model adds Mixture of Attention (MoA) routing to Phi-mini-MoE, then: - ✂️ Pruned 25% of attention heads (kept only important ones) - 🎯 Forced expert specialization (each expert focuses on specific tasks) - ⚡ ~3x…
Access
Public
Updated Jan 31, 2026
Repo size
4.2 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt4.2 MB · 99%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Jan 31, 2026
- Updated
- Jan 31, 2026
- SHA
- ddea8b66
Base models
Model card
From the Hugging Face model README
Phi-mini-MoE + MoA + Pruning + Specialization
What's Special
This model adds Mixture of Attention (MoA) routing to Phi-mini-MoE, then:
- ✂️ Pruned 25% of attention heads (kept only important ones)
- 🎯 Forced expert specialization (each expert focuses on specific tasks)
- ⚡ ~3x faster than OLMoE-1B-7B
Stats
- Base: Phi-mini-MoE (7.6B total, 2.4B active)
- Attention heads: 32 → 24 (pruned 25%)
- Training iterations: 10
- Expert specialization: 16.7%
Files
moa_router.pt- Trained + pruned MoA routertraining_data.json- Self-play examplesexpert_stats.json- Expert specialization profilespruning_stats.json- Which heads were pruned
By
- License
- apache-2.0
- Created
- Jan 31, 2026
- Updated
- Jan 31, 2026