MoE-Study

MoE-Study — Dense vs. Mixture-of-Experts, matched active parameters

Two decoder-only language models trained from scratch under identical conditions, differing in exactly one thing: whether the feed-forward block is a dense MLP or a sparse top-2-of-4 MoE.

Both checkpoints live in this one repo:

The MoE's active-parameter count matches Dense by construction — 2 of 4 experts at half the hidden size means identical compute per token. The MoE only spends more memory for extra capacity.

Full write-up, training code, and evaluation harness: github.com/OliverSundaram/MoE-Study

---

⚠️ These are research artifacts, not usable models

Read this before downloading.

• Trained for one epoch on ~40.7M tokens — neither model is close to converged.
• WikiText word perplexity is 551 (Dense) and 1,378 (MoE). Generations are largely incoherent.
• 0.0% on LAMBADA for both — at the task floor.
• No instruction tuning, no RLHF, no safety filtering of any kind.

They exist to answer one narrow question: at matched active compute and matched budget, does sparsity help? They are not fit for any downstream use.

---

Getting the weights

These are a custom architecture, not a variant of an existing one. The modeling code is not included here, so from_pretrained on this repo alone will not build the model.

Clone the GitHub repo — it carries the model definition and loading instructions, and points back at these subfolders for the weights.

---

Model details

Shared architecture

Both models are the same custom decoder-only transformer:

The one difference

Both models share the same unmodified GPT-2 tokenizer, stored once at the repo root.

---

Training

Identical for both models. Single consumer GPU, no cloud.

Final losses

Dense has the lower loss at every checkpoint.

---

Evaluation

All benchmarks via lm-evaluation-harness on the final checkpoints.

How to read this:

• Dense wins on everything sensitive to raw LLM quality — perplexity, ARC-Easy, PIQA.
• WinoGrande, HellaSwag, and ARC-Challenge are won by MoE, but with such a negligible difference, that they could be considered to have an equal accuracy

Inference speed

Greedy decoding, 32-token prompt → 64 new tokens, 5 trials, 2 warmup, no KV cache.

MoE is ~3.1× slower despite matched active compute — an artifact of unoptimized expert dispatch, not a property of the architecture.

<details> <summary><b>Benchmark charts</b></summary>

</details>

---

Findings

1. Dense won every metric that wasn't already at chance. Most clearly on WikiText perplexity — 551 vs 1,378, a 2.5× gap.

2. The routing math is correct. Active parameters match Dense almost exactly. Matched active compute simply didn't buy matched quality at this budget.

3. Routing stayed balanced. The load-balancing term sat on its theoretical floor, so the MoE's gap is not explained by experts collapsing onto each other.

4. Extra capacity needs extra tokens. The MoE has 38% more parameters but saw the same ~40.7M tokens — likely far too few to train 4 experts per layer, each seeing only a routed fraction of the stream.

Citation

@misc{sundaram2026moestudy,
  author = {Sundaram, Oliver},
  title  = {MoE-Study: Dense vs. Mixture-of-Experts at Matched Active Parameters},
  year   = {2026},
  url    = {https://github.com/OliverSundaram/MoE-Study}
}

Acknowledgments

• lm-evaluation-harness (EleutherAI) — evaluation
• nampdn-ai/tiny-textbooks — training corpus
• Hugging Face transformers — base classes and tokenizer

License

MIT