pythia-160m-ppt-control_nca_steps100-seed1024

sashaboguraev/pythia-160m-ppt-control_nca_steps100-seed1024

pythia-160m-ppt-control_nca_steps100-seed1024 is a machine learning model from sashaboguraev. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

Trained from scratch using the ppt pre-pretraining research framework.

Downloads · 30 days

5% of all-time downloads

All-time downloads

Public

Repo size

8.2 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pt5.3 GB · 62%

At a glance

License: mit
Access: Public
Created: May 12, 2026
Updated: Sep 4, 2026
SHA: 56f92d3e

Base models

EleutherAI/pythia-160m

Model card

From the Hugging Face model README

Pythia-160M Pre-Pretraining: `control_nca_steps100` (seed 1024)

Trained from scratch using the ppt pre-pretraining research framework.

Training Details

Parameter	Value
Base architecture	EleutherAI/pythia-160m (reinitialized)
Regimen	`control_nca_steps100`
Seed	1024
Stage 1 dataset	Shuffled NCA tokens (unstructured control)
Stage 1 steps	100
Stage 2 dataset	OpenWebText
Stage 2 steps	10000
Optimizer	AdamW (lr=1e-3, wd=0.0)
Effective batch size	64
Sequence length	2048

Control Design

Stage 1: NCA tokens with shuffled order. Same token distribution as nca but no sequential structure. This is the unstructured control.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-control_nca_steps100-seed1024")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-control_nca_steps100-seed1024")

Citation

If you use this model, please cite the original pre-pretraining papers:

Papadimitriou & Jurafsky (2020) — tilt-transfer
Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
Lee et al. (2024) — NCA pre-pretraining (danihyunlee)

License: mit
Languages: en
Created: May 12, 2026
Updated: Sep 4, 2026