readout-recipe-control

Readout recipe control models (31M, five matched arms)

Matched 31M Pythia-style (GPT-NeoX architecture) pretraining runs used as the recipe control in Learning to Read Out: Unembedding Dynamics in Language Model Pretraining (main text, Finding 2, and the recipe control appendix). All arms train on the first 10B tokens of the Pythia preshuffled Pile (EleutherAI/pile-standard-pythia-preshuffled, GPT-NeoX-20B tokenizer) read in Pythia's released order, with the same parameter seed (0), fp16 precision, global batch 1024 sequences (2,097,152 tokens/step), weight decay 0.1, peak learning rate 1e-3, and a 10B-token budget (4,769 steps). Each arm changes exactly one recipe factor relative to baseline:

Each arm ships config.json (full training config), metrics.csv (train and validation loss, learning rates, W_U row norm statistics, and final LayerNorm gain norm per logged step), a COMPLETE sentinel, and ckpts/step<N>/{model_fp16.pt,metrics.json} at the 16 checkpoint steps the paper analyzes:

0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, ~2480, 4096, 4769

The step near 2480 is where the first 11-hour job slot ended; the trainer checkpoints on slot stop, so that one step differs slightly between arms (2464 to 2497). The others are the dense early steps, the power-of-two token milestones, and the budget end.

The trainer, tokenizer pipeline, slice builders, and the W_U geometry analysis are in the code release under experiments/ablations/pretraining_recipe_control/ (https://github.com/hematteo/learning-to-read-out; that README lists the exact flags per arm, and docs/DATA.md the data sources and licences). Download into the layout the analysis script expects with hf download hematteo/readout-recipe-control --local-dir "$UM_SSD_ROOT/runs". These are research artifacts for analyzing readout geometry and timing across training, not general purpose language models.

Revision history: the June 2026 upload held a pilot pull of four arms truncated at step 512 (about 1B tokens) with a long_warmup arm at warmup 2860; this revision replaces it with the complete production runs the paper reports.

Citation

@misc{he2026learningtoreadout,
  title  = {Learning to Read Out: Unembedding Dynamics in Language Model Pretraining},
  author = {He, Matteo and Shen, William F. and Iacob, Alex and Jovanovic, Andrej
            and Qiu, Xinchi and Lane, Nicholas D.},
  year   = {2026},
  note   = {Under review. Code: https://github.com/hematteo/learning-to-read-out},
}

MIT. Trained on the first 10B tokens of EleutherAI/pile-standard-pythia-preshuffled; see the Pile's data statement for upstream text provenance.