allenai

dolma2-tokenizer

allenai/dolma2-tokenizer

dolma2-tokenizer is a machine learning model from allenai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

Slightly modified version of cl100kbase that supports Dolma 1.x special tokens (|||PHONENUMBER|||, |||EMAILADDRESS|||, |||IPADDRESS|||) as well as adds extra tokens to fill gaps in tiktoken cl100kbase version.