Downloads · 30 days
355
3% of all-time downloads
gpt2_1558M_final4_hf
karpathy/gpt2_1558M_final4_hf
gpt2_1558M_final4_hf is a text generation model from karpathy. Use it when you need the model to write or continue text. It is set up for transformers.
This is a GPT-2 model trained in llm.c for 330K steps (of 1M batch size) on FineWeb-EDU.
All-time downloads
11K
Public
Parameters
1.6B
3.1 GB on disk
Likes
12
Public
Repo makeup
Click a slice to open those files.
.safetensors3.1 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gpt2
- Access
- Public
- Created
- Jul 12, 2024
- Updated
- Jul 12, 2024
- SHA
- 2a1c4bca
Try a prompt
Model card
From the Hugging Face model README
This is a GPT-2 model trained in llm.c for 330K steps (of 1M batch size) on FineWeb-EDU.
A lot more detailed information is here: https://github.com/karpathy/llm.c/discussions/677 .
This model has a bit of a complicated history. I wanted to train it for 400K steps, i.e. (-x 400000), but it became unstable later in training and exploded around step 330K. Because I was losing my computing quota shortly, I decided to just rewind back to checkpoint 300K, and then instead of going all the way to 400K I started annealing linearly down to 330K. This went without incident and produced this model.
This is the longest I've trained a GPT-2 model for, and it reaches HellaSwag of 62.7 by the end.
- Task
- Text Generation
- Library
- transformers
- Type
- gpt2
- Created
- Jul 12, 2024
- Updated
- Jul 12, 2024