FLAME-MoE-1.7B-10.3B

🧨 FLAME-MoE

This repository contains the model used in the paper FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models.

FLAME-MoE is a fully open Mixture-of-Experts (MoE) language model suite developed by Carnegie Mellon University. It provides a transparent and reproducible research platform for investigating expert routing, model scaling, and training dynamics in sparse architectures. The suite includes seven decoder-only transformer models ranging from 38M to 1.7B active parameters and reflects production-grade MoE setups with 64 experts per MoE layer, top-8 routing, and shared experts.

---

🔍 Model Summary

---

📖 Training Details

• Framework: Megatron-LM with Expert Parallelism (EP=8), Pipeline Parallelism (PP=1)
• Data: Pretrained on DataComp-LM (DCLM)
• Batch Size: 1024
• Sequence Length: 2048
• Optimizer: Adam
• Scheduler: WSD (Warmup + Decay)
• Learning Rate: Max 3e-4, Min 3e-5
• Checkpoints: 10 saved per model across training
• Hardware: 32× NVIDIA H100 GPUs

---

🛠 Intended Use

FLAME-MoE is developed for research purposes only. It supports academic study in:

• Sparse model training dynamics
• Expert routing behavior and specialization
• Scaling laws and compute-optimal design
• Benchmarking and reproducibility in MoE LLMs

It is not intended for commercial deployment or instruction-tuned downstream tasks.

---

📂 Access

All models, training scripts, logs, routing traces, and evaluation pipelines are available at:

🔗 https://github.com/cmu-flame/FLAME-MoE