nanochat-d24-climbmix

h0x91b/nanochat-d24-climbmix

A nanochat model trained from scratch on 8xH100 SXM. Stage: sft, tag d24, step 466.

Config

{
  "sequence_len": 2048,
  "vocab_size": 32768,
  "n_layer": 24,
  "n_head": 12,
  "n_kv_head": 12,
  "n_embd": 1536,
  "window_pattern": "SSSL"
}

Important: this is not a transformers model

These are the native nanochat checkpoint files, not a transformers checkpoint. AutoModel.from_pretrained will not work.

To run it:

git clone https://github.com/karpathy/nanochat && cd nanochat
huggingface-cli download h0x91b/nanochat-d24-climbmix --local-dir "$NANOCHAT_BASE_DIR"
python -m scripts.chat_cli -p "What is the capital of France?"

The tokenizer is included and is required — the weights are meaningless without it.

Resuming training

The optimizer state is included: 8 shards (optim_000466_rank*.pt).

Resume needs all of them and the same world size that wrote them (8 GPUs) — the state is sharded per rank, so a different GPU count cannot load it. Muon momentum and the Adam moments live in there; without them a resume restarts cold and the loss visibly jumps.

torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- \
    --depth=24 --resume-from-step=466