zirui3

llm-multilingual-tokenizer

zirui3/llm-multilingual-tokenizer

llm-multilingual-tokenizer is a machine learning model from zirui3. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.