pvlabs

Chytrej2-Tokenizer

pvlabs/Chytrej2-Tokenizer

Chytrej2-Tokenizer is a machine learning model from pvlabs. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.

A custom 32k vocab BPE tokenizer trained on the FineWeb Edu dataset. Will be used for all Chytrej2 series models.