Skip to content

allenai

dolma2-tokenizer-special-tokens-v3-lc-reasoner

allenai/dolma2-tokenizer-special-tokens-v3-lc-reasoner

dolma2-tokenizer-special-tokens-v3-lc-reasoner is a machine learning model from allenai. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.

Slightly modified version of cl100kbase that supports Dolma 1.x and Dolma 2.x special tokens.

Downloads · 30 days

0

Access

Public

Updated Sep 20, 2025

Repo size

—

Likes

0

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.json6.3 MB · 87%

At a glance

Library
transformers
Access
Public
Created
Sep 18, 2025
Updated
Sep 20, 2025
SHA
f62e5fc2
Library
transformers
Created
Sep 18, 2025
Updated
Sep 20, 2025