Downloads · 30 days
0
toktok-es-8k
marcelo-earth/toktok-es-8k
toktok-es-8k is a machine learning model from marcelo-earth. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
BPE tokenizer trained on Spanish Wikipedia using SentencePiece.
Access
Public
Updated Mar 7, 2026
Repo size
370 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model370 KB · 77%
At a glance
- Access
- Public
- Created
- Mar 7, 2026
- Updated
- Mar 7, 2026
- SHA
- ebd466d9
Model card
From the Hugging Face model README
TokTok Spanish Tokenizer
BPE tokenizer trained on Spanish Wikipedia using SentencePiece.
Details
- Vocab size: 8,000
- Model type: BPE
- Training data: Spanish Wikipedia (100K articles)
- Character coverage: 99.95%
- Byte fallback: enabled
Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.load("tokenizer.model")
text = "El procesamiento de lenguaje natural es fascinante."
tokens = sp.encode_as_pieces(text)
print(tokens)
Why?
English-centric tokenizers (GPT-4, Llama) use 20-40% more tokens on Spanish text. This tokenizer is optimized for Spanish, giving better compression and cheaper inference.
- Created
- Mar 7, 2026
- Updated
- Mar 7, 2026