Downloads · 30 days
0
Bantam-Tokenizer-128k
Theoistic/Bantam-Tokenizer-128k
Bantam-Tokenizer-128k is a machine learning model from Theoistic. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
BPE tokenizer with 128,016 vocabulary entries (128,000 base + 16 special tokens).
Access
Public
Updated May 20, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json9.5 MB · 100%
At a glance
- Access
- Public
- Created
- Nov 16, 2025
- Updated
- May 20, 2026
- SHA
- 45c420b8
Model card
From the Hugging Face model README
Bantam Tokenizer
BPE tokenizer with 128,016 vocabulary entries (128,000 base + 16 special tokens).
Special tokens
| Token | ID | String |
|---|---|---|
unk_token | 128000 | <|UNK|> |
pad_token | 128001 | <|PAD|> |
bos_token | 128002 | <|BOS|> |
eos_token | 128003 | <|EOS|> |
| — | 128004 | <|SYSTEM_START|> |
| — | 128005 | <|SYSTEM_END|> |
| — | 128006 | <|USER_START|> |
| — | 128007 | <|USER_END|> |
| — | 128008 | <|AGENT_START|> |
| — | 128009 | <|AGENT_END|> |
| — | 128010 | <|THINK_START|> |
| — | 128011 | <|THINK_END|> |
| — | 128012 | <|COMPUTE_START|> |
| — | 128013 | <|COMPUTE_END|> |
| — | 128014 | <|IMAGE_START|> |
| — | 128015 | <|IMAGE_END|> |
Config quick-reference
pad_token_id: 128001
bos_token_id: 128002
eos_token_id: 128003
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("path/to/bantam-tokenizer")
- Created
- Nov 16, 2025
- Updated
- May 20, 2026