Downloads · 30 days
25
7% of all-time downloads
nanogpt-3000
ashrafs1/nanogpt-3000
nanogpt-3000 is a text generation model from ashrafs1. Use it when you need the model to write or continue text. The card lists the license as mit.
A 124M-parameter GPT-2-class causal language model built from scratch in PyTorch and trained on FineWeb-10B.
All-time downloads
343
Public
Parameters
162M
649 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors649 MB · 100%
At a glance
- Task
- Text Generation
- License
- mit
- Access
- Public
- Created
- Aug 8, 2026
- Updated
- Aug 15, 2026
- SHA
- 5ad8a32a
Try a prompt
Datasets
Model card
From the Hugging Face model README
nanoGPT-3000
A 124M-parameter GPT-2-class causal language model built from scratch in PyTorch and trained on FineWeb-10B.
This is a pretraining checkpoint (3,000 steps, ~1.57B tokens).
Quick Stats
| Parameters | ~124M |
| Context | 2,048 tokens |
| Vocab | 50,304 (GPT-2 BPE, padded to 128) |
| Position enc | RoPE |
| Norm | RMSNorm + QK-Norm |
| Activation | Squared ReLU |
| Attention | FlashAttention-2 varlen (packed docs) |
| Skips | U-Net style |
| Optimizer | Muon (2D weights) + AdamW (embeddings) |
| Batch size | 524,288 tokens/step |
| Hardware | 2× A100 40GB |
| Time | ~70 min |
| MFU | 70% |
| Throughput | 375K tok/s |
WikiText-2
| Model | Tokens | PPL |
|---|---|---|
| OpenAI GPT-2 | ~40B | 25.2 |
| nanoGPT-3000 | ~1.57B | 40.50 |
How to Use
Repo (training code + inference): https://github.com/shehab-ashraf/nanogpt
Blog (deep dive): https://shehab-ashraf.github.io/posts/nanogpt/
Logs (W&B): https://wandb.ai/ashrafshehab-/nanoGPT
- Task
- Text Generation
- License
- mit
- Languages
- en
- Created
- Aug 8, 2026
- Updated
- Aug 15, 2026