Downloads · 30 days
20
4% of all-time downloads
gpt2-small-fineweb-edu-200m
rockerritesh/gpt2-small-fineweb-edu-200m
gpt2-small-fineweb-edu-200m is a text generation model from rockerritesh. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 124M parameter language model trained from scratch on FineWeb-Edu.
All-time downloads
526
Public
Parameters
124M
498 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors498 MB · 100%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- gpt2
- Access
- Public
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026
- SHA
- 8fddf6f7
Try a prompt
Datasets
Model card
From the Hugging Face model README
gpt2-small-fineweb-edu-200m
A 124M parameter language model trained from scratch on FineWeb-Edu.
Model Description
| Property | Value |
|---|---|
| Architecture | Standard GPT-2 Small architecture |
| Parameters | 124M |
| Layers | 12 |
| Hidden size | 768 |
| Attention heads | 12 |
| Context length | 1,024 tokens |
| Vocab size | 50,257 |
| Final loss | 5.6518 |
| Final perplexity | 284.8 |
Training Details
This model was trained from scratch (random initialization) for research purposes.
Training Data
- Dataset: FineWeb-Edu (sample-10BT split)
- Total tokens: ~200M (streamed)
Training Configuration
| Hyperparameter | Value |
|---|---|
| Batch size | 4 sequences |
| Gradient accumulation | 8 steps |
| Effective batch | 32,768 tokens/step |
| Total steps | 6,103 |
| Learning rate | 6e-4 (cosine decay to 6e-5) |
| Warmup steps | 200 |
| Optimizer | AdamW (beta1=0.9, beta2=0.95) |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | Mixed (AMP fp16) |
| Hardware | NVIDIA T4 GPU (16GB) |
Training Curve
- Initial loss: ~10.9 (random)
- Final loss: 5.6518 (perplexity 284.8)
Limitations
- Not for production use: Trained on only 200M tokens (produces rough text)
- No instruction tuning: Base causal LM, not a chat model
- English only: Trained on English FineWeb-Edu data
Related Models
rockerritesh/gpt2-small-fineweb-edu-200m— GPT-2 Small (124M, 12 layers)rockerritesh/smollm2-135m-fineweb-edu-200m— SmolLM2 (135M, 30 layers, Llama-style)
- Task
- Text Generation
- Type
- gpt2
- License
- apache-2.0
- Languages
- en
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026