gptbert-babycosmofine

GPT-BERT (babycosmofine) - baseline arm

Models for Learning syntax without semantics: Disentangled tiny language models
Ezra Winston and Zico Kolter, ICML 2026

• Paper: OpenReview (PDF)
• Code and reproduction recipes: https://github.com/ezrawinston/sambal

The baseline models: GPT-BERT trained on the un-ablated 10M-word babycosmofine corpus. Their counterparts trained on the SAMBAL-ablated corpus are in gptbert-sambal.

Weights

Reported numbers come from EMA checkpoints throughout, with one exception: the baseline long model's SyntaxGym results (the SyntaxGym column and the per-suite table). Raw (non-EMA) weights were noticed to perform better on SyntaxGym for the baseline model, and the paper reports those raw-weights results. Both baseline weight files are released here; every other result uses EMA weights.

Files

Loading

These are plain state_dict files, not transformers-format checkpoints - AutoModel.from_pretrained will not load them. The architecture, config, and tokenizer all live in the GitHub repo (lm/gpt-bert/); to load and evaluate the checkpoints, follow reproduce/icml2026/README.md.

Model details

• Architecture: GPT-BERT (Charpentier & Samuel, 2024, arXiv:2410.24159), small config - 12 layers, hidden size 384, 6 attention heads, vocabulary 8192.
• Tokenizer: gpt-bert-babylm-small.
• Training data: ltg/babylm-2024-baby-cosmo-fine-10m, revision 5179e7ac0b6be2083ed03444a3a8c3d2c96061a2.
• Short-regime runs: learning rate 0.007, seeds 0/1/2 - the paper's short-regime rows and the targeted-augmentation baselines.
• LoRA runs: rank 32, alpha 16, embeddings also trainable, learning rate 1e-3, checkpoint selected by best dev ppl.

License

These models are released under an MIT license.

Intended use

Research artifacts for studying how lexical ablation of a pretraining corpus affects what a language model learns. These are small models trained on 10M words; they are not instruction-tuned and are not intended for deployment.

Citation

@inproceedings{winston2026syntax,
  title     = {Learning syntax without semantics: Disentangled tiny language models},
  author    = {Winston, Ezra and Kolter, J. Zico},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  series    = {Proceedings of Machine Learning Research},
  volume    = {306},
  year      = {2026},
  publisher = {PMLR},
  url       = {https://openreview.net/forum?id=p7HVrmZwWB}
}