Downloads · 30 days
0
merlin-tokenizer-v0
tsuberim/merlin-tokenizer-v0
merlin-tokenizer-v0 is a machine learning model from tsuberim. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for merlin. The card lists the license as apache-2.0.
BPE tokenizer for Merlin, trained on Python, Bash, Markdown, Stack Exchange Q&A, GitHub commits/issues, and tldr-pages.
Access
Public
Updated Apr 10, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.2 MB · 100%
At a glance
- Library
- merlin
- License
- apache-2.0
- Access
- Public
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026
- SHA
- 7aea5a3f
Model card
From the Hugging Face model README
merlin-tokenizer-v0
BPE tokenizer for Merlin, trained on Python, Bash, Markdown, Stack Exchange Q&A, GitHub commits/issues, and tldr-pages.
Vocab
- 32,016 tokens total: 32,000 BPE + 16 special tokens
- Special tokens:
- IDs 0–13: legacy slots (unused)
- ID 32000:
<|bos|> - ID 32001:
<|eos|> - ID 32002–32013: agent protocol tokens (
<|tool_call|>,<|/tool_call|>, etc.) - ID 32014:
<|done|> - ID 32015:
<|pad|>
Usage
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("def hello(): pass").ids
Note: Will be retrained after agentic trace generation. This is v0 — trained on pretraining corpus only.
- Library
- merlin
- License
- apache-2.0
- Created
- Apr 10, 2026
- Updated
- Apr 10, 2026