babylm2026-strict-gptbert-simper

GPT-BERT + CDI-Curriculum + SimPER A0 (BabyLM 2026 Strict Track)

Model Description

This is the CRADLE system for BabyLM 2026 Strict track (≤100M words, ≤125M parameters). It combines a GPT-BERT hybrid architecture with two complementary innovations:

1. CDI Sentence-Level Lexical Curriculum: Orders pre-training sentences by the Age of Acquisition (AoA) of their vocabulary, using the MacArthur-Bates CDI norms. Early-acquired words (10th-percentile CDI) appear exclusively in Phase 1 (first 10% of training steps), leading to measurable improvements in AoA tracking.

2. SimPER Post-Training (A0): 20,000 preference pairs (15,000 grammar + 5,000 entity-tracking) using the SimPER objective — a hyperparameter-free preference optimization method. This significantly improves Entity Tracking from 40.42 to 46.09 (+5.67 points) while maintaining BLiMP performance.

Compliance

Pre-training corpus: 99.32M words (BabyLM 2026 Strict budget ≤ 100M). SimPER post-training adds ≈0.34M words of entity-tracking text (99.66M total ≤ 100M ✓).

• Official BabyLM sources: CHILDES, Gutenberg, OpenSubtitles (filtered, 858 docs removed), BNC-Spoken; Switchboard removed in full.
• Supplementary: swifte/gutenberg_english, wikimedia/wikipedia

Results (BabyLM 2026 Strict Track)

Architecture

• Architecture: GPT-BERT hybrid (causal GPT head + masked BERT encoder)
• Parameters: ~125M
• Pre-training: 16,372 steps, 4×A100 GPUs, LAMB optimizer
• Post-training: SimPER A0, 300 steps, lr=1e-5, batch size 32

Usage

from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("AliceAndNoob/babylm2026-strict-gptbert-simper")
# See modeling_gpt_bert_eval.py for the custom model class

Citation

@inproceedings{cradle-babylm2026,
  title     = {Developmentally-Inspired Curriculum and Hyperparameter-Free
               Preference Learning for {BabyLM} 2026},
  author    = {Anonymous},
  booktitle = {Proceedings of the BabyLM Challenge 2026},
  year      = {2026},
}