zirui3

llm-zh-tokenizer

zirui3/llm-zh-tokenizer

llm-zh-tokenizer is a machine learning model from zirui3. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-4.0.

Chinese tokenizer trained on Baike by using BPE algorithm