sail

scaling-with-vocab-trained-tokenizers

sail/scaling-with-vocab-trained-tokenizers

scaling-with-vocab-trained-tokenizers is a machine learning model from sail. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

The trained BPE tokenziers with various vocabulary sizes, which uses to study how the vocabulary size affects the performance of language models.