BabySteps_SidBert-100M-mixed

BabySteps SidBert 100M (BabyLM 2026, strict track)

A GPT-BERT hybrid masked/causal language model trained on the official BabyLM 2026 strict corpus (100M words), submitted by team BabySteps to the 2026 BabyLM Challenge.

The central finding behind this model: under a fixed strong architecture at BabyLM scales, optimizer choice and learning-rate tuning outperformed every data-centric intervention we tried: curriculum-ordered corpora, textbook and story corpora, synthetic data generation, and teacher-supervised auxiliary losses. This model is a faithful GPT-BERT reconstruction trained with the Muon optimizer on the unmodified official corpus, trained on two gpus.

Model details

Training data

The official BabyLM 2026 strict corpus (100M words) — no custom, synthetic, or augmented data, and no teacher models.

Results (official BabyLM 2026 evaluation pipeline, MNTP)

SuperGLUE per task: BoolQ 71.8 · MNLI 66.2 · MRPC (F1) 90.7 · MultiRC 67.0 · QQP (F1) 77.4 · RTE 70.5 · WSC 63.5 Finetuning recipe: lr 2e-5, seed 42, batch size 4, 10 epochs.

Checkpoints

Intermediate checkpoints are published as branches following the BabyLM naming convention chck_{N}M, covering the required milestone schedule (1M–10M, 10M–100M, 100M–1000M words seen).

from transformers import AutoModelForMaskedLM, AutoTokenizer

model = AutoModelForMaskedLM.from_pretrained(
    "svsatheesh/BabySteps_SidBert-100M-mixed", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("svsatheesh/BabySteps_SidBert-100M-mixed")

# a specific milestone checkpoint:
early = AutoModelForMaskedLM.from_pretrained(
    "svsatheesh/BabySteps_SidBert-100M-mixed", revision="chck_10M", trust_remote_code=True
)

The model can also be loaded with AutoModelForCausalLM, since GPT-BERT is trained under both objectives.

Reproducibility note

We trained several models with identical configuration and identical seed, differing only in GPU execution. Their scores span a full point of aggregate performance (seven-suite macro average 56.56–57.91, mean 57.31, sd 0.37), with per-suite ranges of 1.8 (BLiMP), 3.6 (BLiMP Supplement), 5.4 (entity tracking), 7.9 (Global PIQA) and 1.7 (SuperGLUE) points. Training is not bit-reproducible under bf16 with non-deterministic CUDA kernels, and small evaluation suites amplify the resulting differences. Point-scale differences between models at this scale should be interpreted with this variance in mind.

We additionally observe that two-GPU data-parallel runs of this configuration score systematically ~1 BLiMP point below single-GPU runs.

Limitations

Trained on 100M words of English only. Not instruction-tuned, not aligned, and not intended for production use — it is a research artifact for studying sample-efficient pretraining. Outputs may be ungrammatical, factually wrong, or reflect biases present in the BabyLM corpus.

Citation

@inproceedings{satheesh2026babysteps,
  title     = {BabySteps at BabyLM 2026: Optimizer Choice and Hyperparameter Tuning
               Beat Data Interventions for Sample-Efficient GPT-BERT Pretraining},
  author    = {Satheesh, Siddhi; Murillo, Jorge; Chen, Yubei},
  booktitle = {Proceedings of the BabyLM Challenge at EMNLP 2026},
  year      = {2026}
}

Built on GPT-BERT (Charpentier & Samuel, 2024) and the Muon optimizer (Jordan et al., 2024).