Downloads · 30 days
9
9% of all-time downloads
pythia-160m-c4-dyck-ppt
gizemyc/pythia-160m-c4-dyck-ppt
pythia-160m-c4-dyck-ppt is a machine learning model from gizemyc. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Pythia-160M pre-pretrained on k-shuffle Dyck, then pretrained on C4.
All-time downloads
105
Public
Parameters
162M
325 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors325 MB · 99%
At a glance
- License
- apache-2.0
- Model type
- gpt_neox
- Access
- Public
- Created
- May 23, 2026
- Updated
- May 23, 2026
- SHA
- b0be15ef
Base models
Model card
From the Hugging Face model README
pythia-160m-c4-dyck-ppt
Pythia-160M pre-pretrained on k-shuffle Dyck, then pretrained on C4.
Part of an experiment reproducing and extending the pruning analysis from "Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases" (Hu et al., 2025, arXiv:2502.19249).
The hypothesis under test: pre-pretraining on English itself yields attention-head circuits as sparse/transferable as pre-pretraining on k-shuffle Dyck.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("gizemyc/pythia-160m-c4-dyck-ppt")
tokenizer = AutoTokenizer.from_pretrained("gizemyc/pythia-160m-c4-dyck-ppt")
Training
- Base architecture:
EleutherAI/pythia-160m(12 layers x 12 heads) - Pretraining data: C4 (English)
- Pre-pretraining (where applicable): k-shuffle Dyck or English/C4
- Type
- gpt_neox
- License
- apache-2.0
- Languages
- en
- Created
- May 23, 2026
- Updated
- May 23, 2026