childes-gpt2-ladder

CHILDES-trained GPT-2 developmental ladder

A ladder of GPT-2-small models trained from scratch on child-directed speech (CHILDES) at increasing amounts of input. It is the language-model side of a children-vs-LMs study of how vocabulary is acquired across "development," and is meant as a research artifact for work on developmental / sample-efficient language modeling.

What's here

180 models = 10 random seeds × 18 nested input budgets (0.5M → 24M words).

development/seed{S}/rung{B}/
  S ∈ {0, 1, 2, 3, 4, 5, 7, 42, 99, 123}
  B ∈ {0.5M, 0.75M, 1M, 1.25M, 1.5M, 1.75M, 2M, 2.5M, 3M, 3.5M, 4M, 5M, 6M, 7M, 8M, 12M, 16M, 24M}

The budgets are nested (each larger budget contains the smaller), and a seed fixes both the initialization and the input-stream order — so each seed is an "individual" accumulating input, i.e. a developmental trajectory. Every model directory is self-contained (weights + tokenizer + config + provenance.json).

Training trajectories (24M rung, 3 seeds)

The development/ models are each run's best-validation endpoint only. For the largest budget (24M words) we also release intermediate checkpoints along training, retrained with the identical configuration for seeds 42, 0 and 123:

trajectory/seed{S}/rung24M/
  step-NNNNNN/          # weights + tokenizer + config + state.json (step, epoch, why, LR)
  log_history.json      # Trainer log: train/val loss per step/epoch, best checkpoint, saved steps
  surprisal.csv         # per-CDI-word mean surprisal (nats) at every saved step

• Checkpoint grid: step 0 (initialization), 60 log-spaced steps up to 12 epochs (67,464 steps), and every epoch end, truncated at the early-stopping point: 68–70 checkpoints per seed. One epoch is ~5,620 steps (batch 8 × 1024 tokens).
• Best vs stop: validation loss bottoms out at epoch 6 (seed 42) or 7 (seeds 0, 123); training stops 4 epochs later (step 56,220 / 61,798 / 61,831). Checkpoints after the best epoch are past the validation minimum (overfitting); log_history.json gives best_model_checkpoint for each seed.
• Learning rate: linear decay is defined over the 40-epoch maximum, so at the stopping point the LR is still ~75% of its 1e-4 peak. state.json records the LR at each checkpoint.
• Fidelity: each seed's best checkpoint reproduces the published 24M ladder model's per-word surprisal at r = 0.984–0.988 (mean within 0.04 nats), with best validation loss within 0.003 nats of the original runs.
• surprisal.csv uses the same CDI contexts and scoring as the ladder analysis (128-token left context, up to 50 contexts per word; weights stored in bf16, scored in fp32).

sub = "trajectory/seed42/rung24M/step-005622"           # end of epoch 1
model = AutoModelForCausalLM.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub, torch_dtype="auto")

Load a model

from transformers import AutoModelForCausalLM, AutoTokenizer
sub = "development/seed42/rung24M"                     # pick any seed/budget
model = AutoModelForCausalLM.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub, torch_dtype="auto")
tok   = AutoTokenizer.from_pretrained("mcxfrank/childes-gpt2-ladder", subfolder=sub)

Use torch_dtype="auto" to preserve the stored bf16 weights (otherwise they upcast to fp32).

Training

• Architecture: GPT-2-small (125.8M params; 12 layers, 768-dim), trained from scratch.
• Tokenizer: GPT2_CHILDES byte-pair encoding (52k vocab), trained on CHILDES (Feng et al. / TinyDialogues).
• Data: English CHILDES (Feng et al. 2024 preparation). Budgets are counted in whitespace words (the developmental currency); each seed shuffles conversations and takes cumulative prefixes to each budget.
• Optimization: up to 40 epochs with early stopping (patience 4) on held-out validation loss; the lowest-validation-loss checkpoint is kept. AdamW, learning rate 1e-4 (linear decay, no warmup), weight decay 0, batch size 8, sequence length 1024, no in-epoch shuffling.
• Precision: stored as bf16 safetensors.

Provenance & reproduction

Each model carries a provenance.json (seed, budget, config). These weights were re-trained from the exact ladder samples used in the analysis; re-trained per-word surprisal reproduces the published values at r = 0.988 (aggregate mean within ~0.04 nats). GPU nondeterminism makes them faithful re-trainings, not bit-identical.

Intended use & limitations

Research artifacts for studying developmental / sample-efficient language modeling and word acquisition. These are child-scale models (≤24M words of input, English child-directed speech only) and are not intended for production use or as general-purpose language models.

Citation

Paper: Children, but not language models, learn words via accelerating accumulation (Frank, in prep). Data & training pipeline: Feng et al. (2024); CHILDES (MacWhinney, TalkBank).

License

To be set by the repository owner. The training data (CHILDES via TalkBank) is subject to TalkBank's terms of use; please respect those when using or redistributing these models.