nanochat-d34-finetuned

nanochat-d34-finetuned

This repo has finetuned (Mid+SFT) checkpoints of karpathy/nanochat-d34, trained using the nanochat framework.

Model Description

• Base Model: karpathy/nanochat-d34 (2.2B parameters, pre-trained on 88B tokens)
• Architecture: GPT-style transformer with depth=34
• Training Pipeline: Mid-training → SFT → RL (optional)
• Hardware: 8x A100-80GB GPUs

Training Details

Base Model (Pre-trained by Karpathy)

• Parameters: 2,217,082,880
• Training tokens: 88,683,315,200 (40x param:token ratio)
• Max sequence length: 2048
• Base CORE score: 0.3382

Dataset

It's same smol-smoltalk which is mentioned in Karpathy OG nanochat discussion

Eval Reports

Full details in report/reports.md

Fine-tuning Pipeline

1. Mid-training: General instruction tuning on SmolTalk, MMLU, GSM8K, Spelling tasks
2. SFT (Supervised Fine-Tuning): Chat-specific training on ARC, GSM8K, SmolTalk
3. RL (Reinforcement Learning): Optional GRPO-style training on GSM8K (if included)

WandB Reports

<img src="https://huggingface.co/pankajmathur/nanochat-d34-finetuned/resolve/main/wand_charts/Screenshot%202025-12-08%20at%208.13.57%E2%80%AFAM.png" width="600" height="600" />

Full Report

Repository Structure

├── tokenizer/
│   ├── tokenizer.pkl          # Tokenizer
│   └── token_bytes.pt         # Token byte mappings
├── mid_checkpoints/d34/       # Mid-training checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatsft_checkpoints_int8/d34/   # SFT checkpoint
│   ├── model_*.pt
│   └── meta_*.json
├── chatrl_checkpoints/d34/    # RL checkpoint (if available)
│   ├── model_*.pt
│   └── meta_*.json
├── report/                    # Evaluation reports
│   └── report.md
└── logs/                      # Training logs

License

MIT License (same as nanochat)

Acknowledgments

• Andrej Karpathy for the nanochat framework and pre-trained base model

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}

• The nanochat community