omarmomen

babylm_tokenizer_32k

omarmomen/babylm_tokenizer_32k

babylm_tokenizer_32k is a machine learning model from omarmomen. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.

This tokenizer is part of the experiments in the published paper at the BabyLM workshop in CoNLL 2023. The paper titled "Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit S…