ohara-moe-0.9B-a91M-base-d12

ohara-moe-0.9B-a91M-base-d12

A fine-grained mixture-of-experts pretrained base model: 911M total parameters, 91M active per token (10x sparse). Trained from scratch with ohara on 2xA100-80GB.

This is a base language model, not a chat model. It continues text; it does not answer questions and never stops on its own. For something that responds, use ohara-moe-0.9B-a91M-chat-d12. Use this one to run your own finetune from a sparse base.

Architecture

64 routed experts + 1 shared expert per layer, top-4 routing, on all 12 layers. The shared expert is always active, so routed experts specialise instead of each relearning the common transformation (DeepSeekMoE's shared-expert isolation).

Routing uses sigmoid gating with normalised weights (softmax over 64 logits drives every weight tiny) and quantile balancing for load balance: the router bias is solved in closed form each optimizer step, so there is no auxiliary loss and no coefficient to tune. Experts dispatch through torch._grouped_mm -- one kernel for all 64, no per-expert Python loop.

Results

Three architectures, identical tokens (1.48B), schedule, learning rates and seed:

4.5% better bits-per-byte than dense, and the gap widened through training (2.2% at step 250, 4.1% at step 2000) rather than saturating. It reached the dense model's final quality in about 29% fewer steps.

Honest caveats. Active FFN FLOPs are 1.094x the dense baseline (5 active experts x 448 vs dense 2048), so roughly 1-2 points of that 4.5% is bought compute rather than architecture. And 911M parameters is 5.6x the memory of the dense model for 1.09x the compute -- MoE wins per token here, not per hour.

For scale: nanochat reaches GPT-2 grade at 0.718 bits/byte using roughly 28x more compute than the 1.4e18 FLOPs spent here. This is a small model.

Files

model.safetensors, config.json, tokenizer. config.json is load-bearing: moe_experts_per_tok and moe_gate_fn leave no trace in any tensor shape, so without them the weights load correctly but route differently than they trained.

Usage

import json
from safetensors.torch import load_file
from ohara.models.llama import Config, Llama

cfg = json.load(open("config.json"))
cfg.pop("architecture"); cfg.pop("iteration")
model = Llama(Config(**cfg))
model.load_state_dict(load_file("model.safetensors"), strict=False)  # rotary rebuilds

Related: dense chat · MoE 8x chat · chat version