lexdec-small-bpe

lexdec-small-bpe is a small, autoregressive llama model featuring character-level tokenization, trained on the 2024/2025 BabyLM dataset. The checkpoints branch contains 19 checkpoints, 10 across the first 10% of pretraining and 9 more for the remaining 9 percent of pretraining.

We used this model to trace the development of linguistic knowledge (word-level, syntax) across pretraining and to compare it to both larger character-level models and comparable subword models:

If you use this model, please cite the following preprint (the final version will be added as soon as it is published):

@misc{bunzeck2025subwordmodelsstruggleword,
      title={Subword models struggle with word learning, but surprisal hides it}, 
      author={Bastian Bunzeck and Sina Zarrieß},
      year={2025},
      eprint={2502.12835},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.12835},}