CompeteSMoE-5.1B

🎉 CompeteSMoE-5.1B

CompeteSMoE-5.1B is a lightweight and integrated variant of the Mixture-of-Experts (MoE) architecture, built upon the Phi-3.5 Mini and SigLIP baselines. This version incorporates the latest CompeteSMoE algorithm enhancements. CompeteSMoE-5.1B demonstrates strong performance across a range of MoE routing strategies, including both standard and star-to-art routing methods. It achieves competitive results compared to recent MoE architectures, such as SharedE-V2 and SharedE-V3, which are inspired by DeepSeek. Despite the architectural innovations of these models especially their use of shared experts CompeteSMoE-5.1B consistently delivers superior or comparable results.

📝 Note: This version of CompeteSMoE-5.1B was trained on a small-scale dataset. 🚧 We're actively working on a stronger, more robust release — coming soon! 🚀 Stay tuned for updates. 💡

Hardware Resources

---

Citation Information

More details can be found in our paper.

If you use CompeteSMoE, please cite it using this BibTeX:

@article{Nguyen2025CompeteSMoES,
  title={CompeteSMoE - Statistically Guaranteed Mixture of Experts Training via Competition},
  author={Nam V. Nguyen and Huy Nguyen and Quang Pham and Van Nguyen and Savitha Ramasamy and Nhat Ho},
  journal={ArXiv},
  year={2025},
  volume={abs/2505.13380},
  url={https://api.semanticscholar.org/CorpusID:278769210}
}