Downloads · 30 days
0
nanochat-d24-speedrun
rkstgr/nanochat-d24-speedrun
nanochat-d24-speedrun is a text generation model from rkstgr. Use it when you need the model to write or continue text.
A 1.4B parameter GPT-2 style model trained from scratch using nanochat on 16×H100 GPUs.
Access
Public
Updated Mar 31, 2026
Repo size
8.5 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pt8.5 GB · 100%
At a glance
- Task
- Text Generation
- Access
- Public
- Created
- Mar 30, 2026
- Updated
- Mar 31, 2026
- SHA
- bb668709
Try a prompt
Datasets
Model card
From the Hugging Face model README
nanochat-d24-speedrun
A 1.4B parameter GPT-2 style model trained from scratch using nanochat on 16×H100 GPUs.
Training
- Architecture: 24-layer Transformer, 1536 hidden dim, 12 heads, 32K vocab
- Training data: 5.8B tokens (ClimbMix), param:data ratio = 8
- Precision: FP8 (tensorwise scaling)
- Hardware: 16× NVIDIA H100
- Throughput: 1.58M tok/sec, 47.6% bf16 MFU
- Pretraining time: 62 minutes
- Total pipeline time: 1h 26m (pretrain + base eval + SFT + chat eval + report)
Results
| Metric | Base | SFT |
|---|---|---|
| Val BPB | 0.715 | - |
| CORE | 0.247 | - |
| ChatCORE | - | 0.360 |
| ARC-Easy | - | 61.3% |
| ARC-Challenge | - | 48.9% |
| MMLU | - | 36.4% |
| HumanEval | - | 11.0% |
| GSM8K | - | 9.8% |
| SpellingBee | - | 99.6% |
Files
base_checkpoints/— Pretrained base model (step 5568)chatsft_checkpoints/— SFT fine-tuned chat model (step 482)tokenizer/— Custom BPE tokenizer (32K vocab)report.md— Full training report
Usage
# Requires the nanochat repo
from nanochat.gpt import GPT, GPTConfig
from nanochat.checkpoint_manager import load_checkpoint
model, metadata = load_checkpoint("path/to/base_checkpoints")
Acknowledgments
Compute resources provided by WestAI. Thanks to the WestAI team for their generous compute contributions.
- Task
- Text Generation
- Created
- Mar 30, 2026
- Updated
- Mar 31, 2026