nanochat-d34-rl

nanochat-d34-rl

This is an RL-trained version of nanochat-d34-finetuned, fine-tuned using GRPO (Group Relative Policy Optimization) on GSM8K math problems.

Model Description

• Base Model: karpathy/nanochat-d34 (2.2B parameters)
• SFT Model: pankajmathur/nanochat-d34-finetuned
• Architecture: GPT-style transformer with depth=34
• Training Pipeline: Pre-training → Mid-training → SFT → RL (GRPO)
• Hardware: 8x NVIDIA A100-SXM4-80GB GPUs

Key Achievement: GSM8K +73.6% Improvement

The RL training significantly boosted math reasoning capabilities while maintaining general performance:

Training Details

RL Configuration (GRPO)

• Run: d34_rl
• Source: SFT checkpoint
• dtype: bfloat16
• device_batch_size: 4
• examples_per_step: 16
• num_samples: 16
• max_new_tokens: 256
• temperature: 1.0
• top_k: 50
• Learning Rates:
  • unembedding_lr: 0.0040
  • embedding_lr: 0.2000
  • matrix_lr: 0.0200
• weight_decay: 0.0
• num_epochs: 1
• Total Steps: 467

Training Metrics (Final)

• Pass@1: 0.2300
• Pass@2: 0.2750
• Pass@3: 0.3275
• Pass@4: 0.3675
• Average Reward: ~0.28
• Average Sequence Length: ~178 tokens

Repository Structure

├── tokenizer/
│   ├── tokenizer.pkl          # Tokenizer
│   └── token_bytes.pt         # Token byte mappings
├── chatrl_checkpoints/d34/    # RL checkpoint
│   ├── model_000466.pt        # Final model weights
│   └── meta_000466.json       # Training metadata
├── report/                    # Evaluation reports
│   └── report.md
└── logs/                      # Training logs

WandB Training Run

<img src="https://huggingface.co/pankajmathur/nanochat-d34-rl/resolve/main/Screenshot%202025-12-08%20at%205.19.32%20PM.png" width="600" height="600" />

Full Report

Related Models

• Base: karpathy/nanochat-d34 - Pre-trained base model
• SFT: pankajmathur/nanochat-d34-finetuned - Mid-training + SFT checkpoint

License

MIT License (same as nanochat)

Acknowledgments

• Andrej Karpathy for the nanochat framework and pre-trained base model
• The nanochat community

@misc{nanochat,
  author = {Andrej Karpathy},
  title = {nanochat: The best ChatGPT that $100 can buy},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/karpathy/nanochat}
}