alisawuffles

superbpe-tokenizer-128k

alisawuffles/superbpe-tokenizer-128k

superbpe-tokenizer-128k is a machine learning model from alisawuffles. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

We developed this SuperBPE tokenizer for model developers who wish to experiment quickly with an off-the-shelf tokenizer in their pretraining pipeline! This is an English SuperBPE tokenizer with a vocab size of 128K,…