Downloads · 30 days
14
6% of all-time downloads
pythia-160m-ppt-c4_ppt_steps100-seed324
sashaboguraev/pythia-160m-ppt-c4_ppt_steps100-seed324
pythia-160m-ppt-c4_ppt_steps100-seed324 is a machine learning model from sashaboguraev. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Trained from scratch using the ppt pre-pretraining research framework.
All-time downloads
217
Public
Parameters
162M
9.7 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt6.5 GB · 60%
At a glance
- License
- mit
- Model type
- gpt_neox
- Access
- Public
- Created
- Jun 9, 2026
- Updated
- Jun 9, 2026
- SHA
- e596369d
Base models
Model card
From the Hugging Face model README
PYTHIA-160M Pre-Pretraining: c4_ppt_steps100 (seed 324)
Trained from scratch using the ppt pre-pretraining research framework.
Training Details
| Parameter | Value |
|---|---|
| Base architecture | EleutherAI/pythia-160m (reinitialized) |
| Regimen | c4_ppt_steps100 |
| Seed | 324 |
| Stage 1 dataset | c4_ppt_steps100 structured data |
| Stage 1 steps | 100 |
| Stage 2 dataset | OpenWebText |
| Stage 2 steps | 10000 |
| Optimizer | AdamW (lr=1e-3, wd=0.0) |
| Effective batch size | 64 |
| Sequence length | 2048 |
Control Design
Stage 1: c4_ppt_steps100 dataset.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-c4_ppt_steps100-seed324")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-c4_ppt_steps100-seed324")
Citation
If you use this model, please cite the original pre-pretraining papers:
- Papadimitriou & Jurafsky (2020) — tilt-transfer
- Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
- Lee et al. (2024) — NCA pre-pretraining (danihyunlee)
- Type
- gpt_neox
- License
- mit
- Languages
- en
- Created
- Jun 9, 2026
- Updated
- Jun 9, 2026