Downloads · 30 days
0
ChrisMcCormick/nanochat-varlen-d24-2026-03-22
nanochat-varlen-d24-2026-03-22 is a machine learning model from ChrisMcCormick. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
nanochat d24 trained with batch size 16, FP8, and flashattention\varlen. Ran on an 8xH100 and outperformed the baseline time by ~1.6%.
Downloads · 30 days
0
Access
Public
Updated Aug 1, 2026
Repo size
8.5 GB
Likes
0
Public
Click a slice to open those files.
.pt8.5 GB · 100%
From the Hugging Face model README
nanochat d24 trained with batch size 16, FP8, and flashattention_varlen. Ran on an 8xH100 and outperformed the baseline time by ~1.6%.
Final results:
step 05567/05568 (99.98%) | loss: 2.388741 | lrm: 0.05 | dt: 1048.30ms | tok/sec: 1,000,264 | bf16_mfu: 60.37 | epoch: 1 pq: 117 rg: 64 | total time: 97.38m | eta: 0.0m
Step 05568 | Validation bpb: 0.724772
Step 05568 | CORE metric: 0.2614
Peak memory usage: 52865.94MiB
Total training time: 97.38m
Minimum validation bpb: 0.724772
import os
from huggingface_hub import hf_hub_download
repo_id = "ChrisMcCormick/nanochat-varlen-d24-2026-03-22"
cache_dir = os.path.expanduser("~/.cache/nanochat")
# Download model checkpoint
hf_hub_download(
repo_id=repo_id,
filename="base_checkpoints/d24_speedrun/model_005568.pt",
local_dir=cache_dir,
)
# Download tokenizer files
hf_hub_download(
repo_id=repo_id,
filename="tokenizer/token_bytes.pt",
local_dir=cache_dir,
)
hf_hub_download(
repo_id=repo_id,
filename="tokenizer/tokenizer.pkl",
local_dir=cache_dir,
)
ChrisMcCormick/decoderstack-d24
is the same d24 architecture pre-trained from scratch by
DecoderStack (handwritten forward/backward, no
autograd, no torch.optim). It ships a ready-made nanochat checkpoint —
base_checkpoints/d24_decoderstack/ with model_005568.pt and 8-rank optimizer shards for
a warm-started SFT.
⚠️ It is deliberately not hosted here, and the two are not interchangeable. That model
was trained on a different 32,768-token vocabulary (the one in
ChrisMcCormick/climbmix_32k_8_170);
31,474 of 32,759 merge ids differ from the tokenizer/ in this repo. Since nanochat
resolves the tokenizer from a single tokenizer/ per base dir, and build_model only
asserts the vocab size — which matches — a base dir holding both checkpoints will
silently feed one of them the wrong vocabulary, producing fluent-looking nonsense rather
than an error. Give each release its own NANOCHAT_BASE_DIR.