mhc-781m-mhc

Realmbird/mhc-781m-mhc

mhc-781m-mhc is a machine learning model from Realmbird. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.

A 781M-parameter GPT-2-shape language model trained on a subset of Dolma v17. This is the mhc variant of a 3-way comparison (residual / mhc / mhc-lite).

Downloads · 30 days

19% of all-time downloads

All-time downloads

Public

Parameters

1.6B

3.1 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors3.1 GB · 100%

At a glance

License: apache-2.0
Access: Public
Created: May 2, 2026
Updated: May 2, 2026
SHA: c21969a3

Datasets

allenai/dolma

Model card

From the Hugging Face model README

mhc-781m-mhc

A 781M-parameter GPT-2-shape language model trained on a subset of Dolma v1_7. This is the mhc variant of a 3-way comparison (residual / mhc / mhc-lite).

Training method: Manifold-Constrained Hyper-Connections (paper-faithful, Sinkhorn-Knopp)
Architecture: ≈ 781M (GPT-2 Large shape, 36×1280, 4 streams when HC enabled)
Tokenizer: GPT-2 BPE (tiktoken gpt2, vocab 50304)
Sequence length: 1024
Optimizer: AdamW (paper used AdamW + Muon; here AdamW only)
Best validation loss: 3.6255 at iter 10000

Training recipe

Recipe follows "Ablate and Rescue" (arxiv 2603.14833) with one deviation: effective batch size is reduced from the paper's 0.5M tokens/step to fit on a single GPU.

	this run	paper
Tokens / step	65,536	524,288
Steps	10,000	10,000+
Tokens seen	~655M	~3.18B
LR / min_lr	3e-4 / 3e-5	3e-4 / 3e-5
Warmup / decay	200 / 10K (cosine)	200 / 10K (cosine)
Weight decay	0.1	0.1
β1, β2	0.9, 0.95	0.9, 0.95
Grad clip	1.0	1.0
Bias	True	True

Load and run

This is not a transformers-native model. The state_dict targets the nanoGPT-style GPT(GPTConfig) class in this repo. To use it:

from huggingface_hub import snapshot_download
import sys, json, torch
from safetensors.torch import load_file

local = snapshot_download(repo_id="Realmbird/mhc-781m-mhc")
sys.path.insert(0, local)
from model import GPT, GPTConfig

with open(f"{local}/config.json") as f:
    cfg = GPTConfig(**json.load(f))
model = GPT(cfg)
sd = load_file(f"{local}/model.safetensors")
model.load_state_dict(sd)
model.eval()

Companion variants

License: apache-2.0
Created: May 2, 2026
Updated: May 2, 2026