Downloads · 30 days
7
12% of all-time downloads
pythia-160m-dyck-ppt-phase1
gizemyc/pythia-160m-dyck-ppt-phase1
pythia-160m-dyck-ppt-phase1 is a machine learning model from gizemyc. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Phase-1 pre-pretraining only: Pythia-160M trained on k-shuffle Dyck (500 steps). Intended as a starting point for C4 pretraining.
All-time downloads
57
Public
Parameters
162M
325 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors325 MB · 99%
At a glance
- License
- apache-2.0
- Model type
- gpt_neox
- Access
- Public
- Created
- May 23, 2026
- Updated
- May 23, 2026
- SHA
- 829fc3bb
Base models
Model card
From the Hugging Face model README
pythia-160m-dyck-ppt-phase1
Phase-1 pre-pretraining only: Pythia-160M trained on k-shuffle Dyck (500 steps). Intended as a starting point for C4 pretraining.
Part of an experiment reproducing and extending the pruning analysis from "Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases" (Hu et al., 2025, arXiv:2502.19249).
The hypothesis under test: pre-pretraining on English itself yields attention-head circuits as sparse/transferable as pre-pretraining on k-shuffle Dyck.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("gizemyc/pythia-160m-dyck-ppt-phase1")
tokenizer = AutoTokenizer.from_pretrained("gizemyc/pythia-160m-dyck-ppt-phase1")
Training
- Base architecture:
EleutherAI/pythia-160m(12 layers x 12 heads) - Pretraining data: C4 (English)
- Pre-pretraining (where applicable): k-shuffle Dyck or English/C4
- Type
- gpt_neox
- License
- apache-2.0
- Languages
- en
- Created
- May 23, 2026
- Updated
- May 23, 2026