mhc-781m-residual

mhc-781m-residual

A 781M-parameter GPT-2-shape language model trained on a subset of Dolma v1_7. This is the residual variant of a 3-way comparison (residual / mhc / mhc-lite).

• Training method: vanilla residual (no hyper-connections)
• Architecture: ≈ 781M (GPT-2 Large shape, 36×1280, 4 streams when HC enabled)
• Tokenizer: GPT-2 BPE (tiktoken gpt2, vocab 50304)
• Sequence length: 1024
• Optimizer: AdamW (paper used AdamW + Muon; here AdamW only)
• Best validation loss: 3.6156 at iter 9500

Training recipe

Recipe follows "Ablate and Rescue" (arxiv 2603.14833) with one deviation: effective batch size is reduced from the paper's 0.5 M tokens/step to fit on a single A100-40GB.

Load and run

This is not a transformers-native model. The state_dict targets the nanoGPT-style GPT(GPTConfig) class in this repo. To use it:

from huggingface_hub import snapshot_download
import sys, json, torch
from safetensors.torch import load_file

local = snapshot_download(repo_id="Realmbird/mhc-781m-residual")
sys.path.insert(0, local)  # so `model` and `hyper_conn` import
from model import GPT, GPTConfig

with open(f"{local}/config.json") as f:
    cfg = GPTConfig(**json.load(f))
model = GPT(cfg)
sd = load_file(f"{local}/model.safetensors")
model.load_state_dict(sd)
model.eval()

Companion variants

• Realmbird/mhc-781m-residual
• Realmbird/mhc-781m-mhc
• Realmbird/mhc-781m-mhc-lite