BabySteps_MurphysLaw-10M-mixed

BabySteps MurphysLaw 10M (mixed)

BabyLM 2026 Challenge — strict-small track submission (Team BabySteps)

A GPT-BERT hybrid masked/causal language model trained on the official BabyLM 2026 strict-small corpus (10M words), using the AdaMuon optimizer with a tuned learning rate and tail-weight averaging over the final 20% of training.

Our central finding: after data-centric experimentation (curriculum-ordered corpora, story and textbook corpora, synthetic data generation, teacher-supervised objectives), none of those interventions beat a faithful GPT-BERT reconstruction trained on the official corpus with a Muon-family optimizer and a careful learning-rate sweep.

Model details

Training

Tail averaging left zero-shot scores essentially unchanged (±0.3 BLiMP) but improved finetuned SuperGLUE by ≈1.6 points on the development twin of this configuration.

Checkpoints

Intermediate checkpoints are published as branches following the challenge's naming convention, covering the strict-small milestone schedule (1M–10M words in 1M steps, then 10M–100M in 10M steps — 19 checkpoints total):

from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained(
    "svsatheesh/BabySteps_MurphysLaw-10M-mixed",
    revision="chck_7M",           # any of chck_1M ... chck_100M
    trust_remote_code=True,
)

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("svsatheesh/BabySteps_MurphysLaw-10M-mixed")
model = AutoModelForMaskedLM.from_pretrained(
    "svsatheesh/BabySteps_MurphysLaw-10M-mixed", trust_remote_code=True
)

trust_remote_code=True is required — the GPT-BERT modeling code ships with the repo (modeling_gpt_bert.py, configuration_gpt_bert.py). The model can be loaded as AutoModelForMaskedLM or AutoModelForCausalLM; all reported results use MNTP (masked-next-token-prediction) evaluation, which outperformed pure-causal and pure-masked evaluation on the aggregate in our sweeps.

Results (official BabyLM 2026 evaluation pipeline, MNTP)

SuperGLUE finetuning used the official pipeline at learning rate 8–9e-5, seed 44. SuperGLUE convention: F1 for MRPC and QQP, accuracy elsewhere.

Training data

Only the official BabyLM 2026 strict-small corpus was used — no custom data, no synthetic data augmentation, no teacher models, no external pretraining. The model is fully compliant with the strict-small track's data budget.

Limitations

This is a deliberately small model trained on a developmentally plausible data budget. It is a research artifact for studying sample-efficient pretraining, not a general-purpose language model: it has limited world knowledge, produces low-quality free-form generation, and inherits whatever biases exist in the BabyLM corpus (CHILDES, Project Gutenberg, OpenSubtitles, Simple Wikipedia, BNC, Switchboard). It should not be deployed in any user-facing application.

Reported scores are from a single training run.

Citation

@inproceedings{satheesh2026babysteps,
  title  = {BabySteps at BabyLM 2026: Optimizer Choice and Hyperparameter Tuning
            Beat Data Interventions for Sample-Efficient GPT-BERT Pretraining},
  author = {Satheesh, Siddhi; Murillo, Jorge; Chen, Yubei},
  booktitle = {Proceedings of the BabyLM Challenge 2026},
  year   = {2026}
}

Built on GPT-BERT (Charpentier & Samuel, 2024), Muon (Jordan et al., 2024), and AdaMuon (Si et al., 2025).