pietrolesci

tokenisers

pietrolesci/tokenisers

tokenisers is a machine learning model from pietrolesci. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

Tokenisers trained on the MiniPile. The rawtokenisers folder contains the original tokenisers trained with a vocabulary size of 320k. Then, each folder is a transformers-compatible tokeniser of a smaller size.