Downloads · 30 days
17
3% of all-time downloads
smollm2-135m-fineweb-edu-200m
rockerritesh/smollm2-135m-fineweb-edu-200m
smollm2-135m-fineweb-edu-200m is a text generation model from rockerritesh. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 135M parameter language model trained from scratch on FineWeb-Edu.
All-time downloads
495
Public
Parameters
135M
538 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors538 MB · 99%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- llama
- Access
- Public
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026
- SHA
- 127fef60
Try a prompt
Datasets
Model card
From the Hugging Face model README
smollm2-135m-fineweb-edu-200m
A 135M parameter language model trained from scratch on FineWeb-Edu.
Model Description
| Property | Value |
|---|---|
| Architecture | Deep & narrow Llama-style architecture with Grouped Query Attention |
| Parameters | 135M |
| Layers | 30 |
| Hidden size | 576 |
| Attention heads | 9 (3 KV, GQA) |
| Context length | 1,024 tokens |
| Vocab size | 49,152 |
| Final loss | 5.4439 |
| Final perplexity | 231.3 |
Training Details
This model was trained from scratch (random initialization) for research purposes.
Training Data
- Dataset: FineWeb-Edu (sample-10BT split)
- Total tokens: ~200M (streamed)
Training Configuration
| Hyperparameter | Value |
|---|---|
| Batch size | 4 sequences |
| Gradient accumulation | 8 steps |
| Effective batch | 32,768 tokens/step |
| Total steps | 6,103 |
| Learning rate | 6e-4 (cosine decay to 6e-5) |
| Warmup steps | 200 |
| Optimizer | AdamW (beta1=0.9, beta2=0.95) |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | Mixed (AMP fp16) |
| Hardware | NVIDIA T4 GPU (16GB) |
Training Curve
- Initial loss: ~10.9 (random)
- Final loss: 5.4439 (perplexity 231.3)
Limitations
- Not for production use: Trained on only 200M tokens (produces rough text)
- No instruction tuning: Base causal LM, not a chat model
- English only: Trained on English FineWeb-Edu data
Related Models
rockerritesh/gpt2-small-fineweb-edu-200m— GPT-2 Small (124M, 12 layers)rockerritesh/smollm2-135m-fineweb-edu-200m— SmolLM2 (135M, 30 layers, Llama-style)
- Task
- Text Generation
- Type
- llama
- License
- apache-2.0
- Languages
- en
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026