MOME-NanoChat-D24-100B

MOME-NanoChat-D24-100B

Raw NanoChat checkpoints for the three d24 models trained for 100,000 optimizer steps with a 1,048,576-token global batch (104,857,600,000 training tokens). These files are native PyTorch checkpoints, not Transformers-format models.

Models

All runs use the shared 32k tokenizer and the post-NeurIPS Recipe C settings on ClimbMix. The effective arguments are retained in each meta_100000.json and in configs/d24_100b.json.

Reproducibility

• Training code: commit c85060f297840f29c470028fa2c886110c70a3e1 (recipe-upgrade-denseve-d24).
• Data: ClimbMix training shards 0--2499; validation shard 6542.
• Hardware/runtime: eight H100 GPUs and Flash Attention 3.
• Seed: 42.
• Context length: 2,048.

The organized reproduction branch provides:

bash experiments/reproduce/d24_100b.sh --plan
bash experiments/reproduce/d24_100b.sh

Loading

Use the pinned NanoChat code and the matching metadata:

import torch

state_dict = torch.load("model_100000.pt", map_location="cpu", weights_only=True)

Model construction is defined by model_config in the adjacent metadata file. See SHA256SUMS before using a downloaded checkpoint.

Checkpoint hashes

Intended use

These checkpoints are research artifacts for reproducing the reported pretraining and evaluation results. They have not been instruction-tuned or safety-tuned for deployment.