Downloads · 30 days
70
6% of all-time downloads
fineweb_pretrained_model_37M
zyberg2091/fineweb_pretrained_model_37M
fineweb_pretrained_model_37M is a text generation model from zyberg2091. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
A 37.8M-parameter decoder-only language model implemented and pretrained from scratch in PyTorch.
All-time downloads
1.2K
Public
Parameters
37.8M
151 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors151 MB · 99%
At a glance
- Task
- Text Generation
- License
- apache-2.0
- Model type
- localllm
- Access
- Public
- Created
- Mar 8, 2026
- Updated
- Aug 30, 2026
- SHA
- 97c61699
Try a prompt
Model card
From the Hugging Face model README
FineWeb Pretrained Model - 37.8M Parameters
A 37.8M-parameter decoder-only language model implemented and pretrained from scratch in PyTorch.
Model summary
- Parameters: 37.8M
- Layers: 6
- Hidden dimension: 512
- Attention heads: 8
- KV heads: 4
- Context length: 256
- Vocabulary: 20,000
- Architecture: grouped-query attention, RoPE, pre-norm transformer
- Training data: 250M-token FineWeb slice
- Training: approximately 61,000 steps and 500M token passes
- Precision: float32
Intended purpose
This model is a research and educational artifact for studying small-language-model pretraining, architecture implementation, and parameter allocation. It is not intended as a production assistant.
Loading
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "zyberg2091/fineweb_pretrained_model_37M"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
)
- Task
- Text Generation
- Type
- localllm
- License
- apache-2.0
- Languages
- en
- Created
- Mar 8, 2026
- Updated
- Aug 30, 2026