nanochat-d24-chat-champion

nanochat-d24-chat-champion (provisional)

The strongest chat checkpoint produced on a single RTX 4070 in this project — and it did not replicate.

Run tag: d24_r32_adamw_partial_fr20_mixv2_2026-03-29_1040_s768_gc, step 300. Adapted from Marcolini/nanochat-d24-base-champion @ 820,230.

📊 SFT runs on W&B · 📄 Experiment log and 14 failed branches · 🧪 The data mixes that lost

Why "provisional"

This is the best single confirmed run on the chat branch. It is published as provisional, not promoted, because it failed its own reproducibility rule:

• direct replication of the same recipe failed the quick gate
• a deterministic 3-seed sweep failed 0 of 3
• mixv3, mixv4, a two-stage curriculum, stage-B boosters and a curated-reasoning branch all failed to beat it

At this scale a single run sits inside seed noise. The seed sweep is the evidence, and it says this number is not reliably reachable. Treat the metrics below as one draw, not an expectation.

Evaluation

1000-problem confirm pack.

The strongest reasoning result on the project belongs to a different, unpromoted run — reasoning_manual_v1_seed42 at 6.70% GSM8K pass@8 and 27.60% MMLU — whose companion seed failed the gate outright.

Recipe

Note on data: this checkpoint was trained on the built-in reasoning_focus_v2 mixture, not on the JSONL files in the companion dataset. Those are the eight later mixes that tried to beat this recipe and failed. They are published so the comparison can be checked.

Full LoRA vs partial-fine-tune ablations, the OOM ladder that made partial FT fit in 12 GB, and every failed branch are in report_v3.md.

Limitations

• 27.40% MMLU is 2.4 points above random guessing. This model does not have meaningful general knowledge.
• 4.60% GSM8K pass@8 means it fails ~95% of grade-school word problems given eight attempts. Reasoning is essentially absent at this scale.
• SpellingBee 0.39% — character-level tasks are beyond it.
• No alignment or safety training beyond what the SFT mixture provides. It will produce confident fabrication.
• English only, inheriting FineWeb-Edu's coverage and biases.
• Not a transformers model — see below.

Intended for research on consumer-GPU training, not for deployment.

Usage

Not loadable with AutoModelForCausalLM. The architecture uses rotary embeddings, QK-norm, relu² MLP, untied embeddings, GQA, ResFormer value embeddings, and per-layer resid_lambdas / x0_lambdas scalars, so it needs the nanochat GPT class:

git clone https://github.com/Bl4ckd09/nanochat-on-rtx4070
cd nanochat-on-rtx4070
pip install torch safetensors huggingface_hub

python -c "from huggingface_hub import snapshot_download as d; d('Marcolini/nanochat-d24-chat-champion', local_dir='ckpt')"
cd ckpt && PYTHONPATH=.. python load_example.py

A matching 32,768-vocab tokenizer is not bundled; bring the one from the fork.

Citation

@misc{nanochat_rtx4070_chat_2026,
  author = {Sun},
  title  = {nanochat-d24-chat-champion: a provisional chat checkpoint from one RTX 4070},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/Marcolini/nanochat-d24-chat-champion}
}

MIT, as upstream.