EspBPE-49K

📌 Overview

EspBPE-49K is a custom, hyper-optimized 49,152-vocabulary Byte-Level Byte Pair Encoding (BPE) tokenizer built specifically for modern Spanish Large Language Models (LLMs).

Trained across a massive multi-domain Spanish dataset mixture, EspBPE-49K achieves an extraordinary compression ratio (~4.8 to 5.1 characters per token), preserving contextual length and embedding space efficiency across formal encyclopedic text, technical documentation, sports commentary, and regional internet conversations.

---

⚡ Quickstart

You can load EspBPE-49K directly using the Hugging Face transformers library:

from transformers import AutoTokenizer

# Load EspBPE-49K directly from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("DinoResearch/EspBPE-49K")

# Sample text
text = "Estamos entrenando un tokenizer en español super rápido y eficiente con 49,152 de vocabulario! 🚀"

# Encode text into token IDs
input_ids = tokenizer.encode(text)
tokens = tokenizer.convert_ids_to_tokens(input_ids)

print(f"Total Tokens: {len(input_ids)}")
print("Tokens:", tokens)
print("Decoded:", tokenizer.decode(input_ids))

---

📊 Performance & Compression Ratio

---

🗃️ Training Dataset Mixture

EspBPE-49K was trained on a carefully balanced multi-source Spanish corpus:

• FineWeb-2 (spa_Latn): Modern, high-quality web crawl text.
• Spanish Wikipedia (20231101.es): Encyclopedic knowledge baseline.
• CulturaX (es): Filtered, clean multilingual web data (OSCAR + mC4).
• Spanish Tweets (pysentimiento): Informal dialogue, regional accents, and modern social media slang.
• PleIAs Spanish PD-Books: Classic literature and historical Spanish prose.

---

🌟 Key Highlights & Features

• Single-Token Complex Vocabulary: Long formal words like inconstitucionalidad, extraordinariamente, descentralización, afortunadamente, and Rehabilitación map directly to single Token IDs.
• Native Accent & UTF-8 Support: Seamless byte-level decoding for accented characters (á, é, í, ó, ú, ñ) without producing <unk> errors.
• Native Emoji Primitives: High-frequency multi-byte emojis (e.g., 🤪, 😌, 🇵🇷, 🟢) are learned as native single-token lookup entries.
• Regional & Technical Fluency: Optimized subword roots for tech terms (javascript, tensor, Unix, coaxial) alongside regional slang across Spanish-speaking countries.

---

📜 Model Card Details

• Vocabulary Size: 49,152 (49k)
• Model Type: ByteLevel BPE
• Space Prefix: Ġ (ByteLevel space representation)
• Organization: DinoResearch## ⚔️ Benchmark: EspBPE-49K vs. OpenAI o200k_base (GPT-4o)

To evaluate real-world web performance, EspBPE-49K was benchmarked head-to-head against OpenAI's o200k_base tokenizer (used in GPT-4o) across 500 unfiltered Spanish web documents (2,292,109 raw characters) from the FineWeb-2 (spa_Latn) dataset stream.

🏆 500-Text FineWeb-2 Benchmark Results

🧠 Benchmark Key Takeaways

• Massive Parameter Savings: EspBPE-49K achieves superior compression using less than 1/4th of o200k_base's vocabulary size, saving tens of millions of embedding parameters during pretraining.
• 6.90% Token Reduction: Encodes identical Spanish text into 6.90% fewer overall tokens, directly reducing memory requirements and accelerating autoregressive pretraining speed.
• Extended Context Window: Higher token efficiency means more raw text fits into a fixed context length (e.g., 2048 / 4096 tokens).

---