microme-220m

MicroMe-220M

A 217M-parameter deep-thin GPT trained from scratch on a single RTX 4060 laptop (8 GB), the scaled-up capstone of the MicroMe project. It applies the recipe that a controlled 3-seed ablation picked out at small scale: tied input/output embeddings (a parameter-efficiency win at this vocab-to-size ratio) and the Muon optimizer, with gradient checkpointing to fit 217M on 8 GB.

Trained on 2B FineWeb-Edu tokens in ~38 hours. It reaches a lower validation loss (2.896) than the 125M base (2.969) on one-third less data, and improves on it across all four benchmarks.

Benchmarks (0-shot, lm-evaluation-harness)

Beats GPT-2-124M / Pythia-160m on ARC-Easy and HellaSwag, on far less training data. LAMBADA stays the weak spot (small 24k custom vocab, short training run). This is a base model, not chat-tuned; it is not a reliable knowledge store on its own (pair with retrieval for facts).

Architecture

28 layers, d=768, 12 heads, ctx 1024, vocab 24,576. RMSNorm, RoPE, QK-norm, ReLU2 MLP, tied embeddings, logit softcap. Details and training code: github.com/ankit-rawani/microme.

Files

Load

import torch, importlib.util
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

d = hf_hub_download("Ankitgdes/microme-220m", "model.py").rsplit("/", 1)[0]
spec = importlib.util.spec_from_file_location("mm", f"{d}/model.py")
mm = importlib.util.module_from_spec(spec); spec.loader.exec_module(mm)
model = mm.GPT(mm.PRESETS["mini_220m"]).eval()
model.load_state_dict(load_file(hf_hub_download("Ankitgdes/microme-220m", "model.safetensors")))

License

Apache-2.0. Trained on FineWeb-Edu (ODC-By).