llama-lang-adapt

yo-extend-tokenizer

llama-lang-adapt/yo-extend-tokenizer

yo-extend-tokenizer is a machine learning model from llama-lang-adapt. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

Llama tokenizer with extended vocab: - 32000 Llama tokens - ~15000 Yoruba tokens