anlp-a2-p1-moe-top2-wide

ANLP A2 checkpoint: moe-top2-wide

MoE: 4 experts (2x wide), top-2 -- active params == dense

• Part 1 of the Advanced NLP Assignment 2 (Monsoon 2026).
• Task: vi/ja -> en translation (decoder-only LM over <lang> src <en> tgt)
• Token budget: 145,020,416 tokens (vi/ja -> en, belumind/en-vi-ja-curated-500k-triplets)
• Model: 35.7M-parameter decoder-only transformer (D=512, 6 layers, 8 heads, vocab 32768, tied embedding), final checkpoint (end of token budget).
• Total params: 48,253,440 | Active params: 35,670,528

Files

• model.pt -- {"model": <state_dict>, "config": <TransformerConfig dict>}
• config.json -- the TransformerConfig
• tokenizer.json -- byte-level BPE (v32768) trained on the training split
• README.md -- this file

Loading (with the assignment repo checked out)

from src.part1.model import TransformerLM
from src.part1.data import load_tokenizer

model = TransformerLM.load("part1/moe_top2_wide/final", device="cuda")
tokenizer = load_tokenizer("tokenizer")  # or: from this repo's tokenizer.json