Downloads · 30 days
0
zoof-tokenizer
Jiraya/zoof-tokenizer
zoof-tokenizer is a text generation model from Jiraya. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
This is the custom Byte-Pair Encoding (BPE) tokenizer built for the zoof language model family. It was trained from scratch to efficiently handle English text and code.
Access
Public
Updated Jan 5, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json5.7 MB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Dec 8, 2025
- Updated
- Jan 5, 2026
- SHA
- e0152eca
Try a prompt
Model card
From the Hugging Face model README
Zoof Tokenizer
This is the custom Byte-Pair Encoding (BPE) tokenizer built for the zoof language model family. It was trained from scratch to efficiently handle English text and code.
Most recent zoof model: zoof-v1.2-394M-chat.
Model Details
- Vocabulary Size: 49152
- Type: Byte-Pair Encoding (BPE)
- Language: English
- Intended Use: Tokenization for the Zoof model family.
Usage
You can load this tokenizer directly with the transformers library:
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("Jiraya/zoof-tokenizer")
text = "Hello, world!"
tokens = tokenizer.encode(text)
decoded = tokenizer.decode(tokens)
print(f"Tokens: {tokens}")
print(f"Decoded: {decoded}")
- Task
- Text Generation
- Library
- transformers
- License
- mit
- Languages
- en
- Created
- Dec 8, 2025
- Updated
- Jan 5, 2026