Downloads · 30 days
0
gpt2-124M
ShaohuaDong/gpt2-124M
gpt2-124M is a machine learning model from ShaohuaDong. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
A 124M-parameter GPT-2 model was trained on the 10B fineweb-edu dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) using Karpathy's llm library (https://github.com/karpathy/build-nanogpt). Training to…
Access
Public
Updated Oct 30, 2024
Repo size
498 MB
Likes
3
Public
Repo makeup
Click a slice to open those files.
.pt498 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Oct 30, 2024
- Updated
- Oct 30, 2024
- SHA
- 3b4f44de
Model card
From the Hugging Face model README
A 124M-parameter GPT-2 model was trained on the 10B fineweb-edu dataset (https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) using Karpathy's llm library (https://github.com/karpathy/build-nanogpt). Training took 3 hours on 4 H100(80G) GPU, yielding the following graphs:
Settings:
Model Parameters: 124M
Tokens: 10B
Batch Size: 48
Max Sequence Length: 1024
Total Batches: 196608
Warmup Steps: 1906
Max Steps: 50862
GPUs: 4
GPU Memory Usage: 62567 MB
Training Time: 3 hours (12 GPU hours)
- License
- mit
- Created
- Oct 30, 2024
- Updated
- Oct 30, 2024