pocket-moe

Pocket MoE

Pocket MoE is a compact top-2 mixture-of-experts classifier. A shared image encoder feeds four specialist MLPs while a learned router activates two experts for each example.

The benchmark compares an MoE and a parameter-matched dense MLP on the same split, then reports expert utilization, routing entropy, and each digit's dominant expert.

Reproduce

uv run python projects/tiny-vision-foundry/prepare_data.py
uv run python projects/pocket-moe/train.py

Sparse routing here demonstrates conditional computation; it does not reduce the stored checkpoint size because all four experts remain available.

Verified results

The MoE activates 34.3% fewer parameters per example than the dense control without losing accuracy. Experts specialized strongly: digit 0 routed 86.5% to expert 3, digit 2 routed 83.1% to expert 2, and digit 7 routed 90.9% to expert 0.

Utilization was not perfectly balanced: expert shares were 44.2%, 12.4%, 19.6%, and 23.8%, for a coefficient of variation of 0.473. The stored MoE still includes all four experts, so conditional compute should not be confused with checkpoint compression.