moe-causal-top2-generalist

moe-causal-top2-generalist

BabyLM 2026 Strict-Small submission (NTUA). A shared-generalist Mixture-of-Experts language model: decoder-only causal MoE — hidden 512, 12 layers, all-MoE with an always-on generalist FFN (size 1024) + 15 routed specialists (size 256), top-2 softmax routing.

• Training data: BabyLM-community/BabyLM-2026-Strict-Small (≤100M words, counting repeated exposures, per CfP §4.2). Uniform ~10 passes via bin-packing (each segment once per epoch).
• Objective: causal (next-token) LM. Evaluate with the causal backend.
• Intermediate checkpoints: available as branches chck_1M … chck_100M (words seen), per the BabyLM checkpoint convention.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("JimK28/moe-causal-top2-generalist", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("JimK28/moe-causal-top2-generalist", trust_remote_code=True)

Custom architecture — trust_remote_code=True is required (config auto_map points at modeling_moe_causal). Trained with the NTUA BabyLM pipeline (https://github.com/emhadzi/BabyLM).