miguelcsx

tolm-wordlevel-16k-tokenizer

miguelcsx/tolm-wordlevel-16k-tokenizer

tolm-wordlevel-16k-tokenizer is a machine learning model from miguelcsx. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for tokenizers. The card lists the license as other.

Whitespace-split word-level tokenizer with vocabulary size 16000, trained on the BabyLM Strict-Small cleaned corpus.