gemma3moe-16x4-270m-starter

gemma3moe-16x4-270m-starter

A ready-made Mixture-of-Experts initialization — 16 experts, top-4 — sparse-upcycled from google/gemma-3-270m. This is a starting point for your own training / fine-tuning, not a trained model: it is the standard copy-init upcycle, so at step 0 it is mathematically identical to the dense Gemma-3-270m. Load it, then train.

⚠️ Untrained. Out of the box it behaves exactly like gemma-3-270m (verified: max logit diff 0.000e+00). Its value is as a clean MoE init with more capacity to train into.

• 🧩 Architecture: gemma3moe — Gemma-3 with each dense FFN replaced by a router + 16 Gemma3MLP experts (softmax top-4, renormalized).
• 💾 Size: ~1.33B total / ~481M active (top-4). Full 262k vocabulary (not trimmed).
• ⚡ CPU-friendly: active params — not total — drive latency; runs on llama.cpp.
• 🔁 Self-contained: ships modeling_gemma3_moe.py + auto_map, so it reloads with trust_remote_code=True.

Why start from an upcycled MoE?

Sparse upcycling initializes each expert from the pretrained dense FFN, so you keep all of Gemma-3-270m's learned knowledge while adding capacity (16× the FFN parameters) that training can specialize. Copy-init makes the MoE exactly equal to the dense model at step 0 — a stable, well-behaved starting point, strictly better than training a randomly-initialized MoE from scratch.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "tardellirs/gemma3moe-16x4-270m-starter"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16)

# It's an INIT — fine-tune it on your data before using for generation.

Choose a different experts/top-k or a trimmed vocabulary, and serve on CPU via llama.cpp, using the open-source toolkit that built this checkpoint:

➡️ github.com/tardellirs/gemma3moe-builder

# reproduce this checkpoint
python -m gemma3moe.upcycle --src google/gemma-3-270m --out ./starter --num-experts 16 --top-k 4
# fine-tune
python -m gemma3moe.train --init ./starter --train train.jsonl --val val.jsonl --out ./trained

Details

• Router initialized small but non-zero (std=0.02) so gradients can differentiate the experts during training; all 16 experts are copies of the dense FFN at init.
• Add a Switch-style load-balance aux loss (+ z-loss) when you train (see the toolkit's train.py and docs/NOTES.md).
• 18 layers, hidden 640, expert intermediate 2048.

License

Weights derive from Gemma-3 and are subject to the Gemma Terms of Use. Built with gemma3moe-builder (Apache-2.0).