Downloads · 30 days
2
1% of all-time downloads
gpt-hilberg-5M
AdrianRasoOnHF/gpt-hilberg-5M
gpt-hilberg-5M is a machine learning model from AdrianRasoOnHF. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The present is a 5M-parameter GPT autoregressive language model trained on the July 20, 2025 English Wikipedia dump for experiments on entropy scaling and Hilberg conjecture. For more information on this, you can chec…
All-time downloads
180
Public
Repo size
35.1 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin35.1 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Dec 7, 2025
- Updated
- Dec 7, 2025
- SHA
- 297fd82d
Model card
From the Hugging Face model README
GPT-Hilberg-5M
The present is a 5M-parameter GPT autoregressive language model trained on the July 20, 2025 English Wikipedia dump for experiments on entropy scaling and Hilberg conjecture. For more information on this, you can check here. Dataset available here.
This model is part of the following suite:
- GPT-Hilberg-1M (128 ctx)
- GPT-Hilberg-5M (512 ctx)
- GPT-Hilberg-10M (1024 ctx)
Architecture details
- Parameters: ~5M
- Layers: 10
- Model dimension: 128
- Attention heads: 8
- FF dimension: 512
- Context length: 512
- Vocabulary size: 16,000
Training details
- Dataset: English Wikipedia (2025-07 dump)
- Tokenizer: SentencePiece BPE
- Purpose: Autogregression
- Optimizer: AdamW
- License
- mit
- Created
- Dec 7, 2025
- Updated
- Dec 7, 2025