exp-vocab-64k

exp_vocab_64k

A production-grade, general-purpose Byte-Level Byte Pair Encoding (BPE) Tokenizer built from scratch for modern generative language models.

🌟 Highlights

• Zero Out-Of-Vocabulary (0.00% UNK Rate): Full byte fallback ensures every single arbitrary byte sequence (0–255) is representable.
• 100% Lossless Roundtrip Fidelity: decode(encode(text)) == text across arbitrary code, LaTeX, emojis, whitespaces, tabs, and Unicode scripts.
• Optimized Subword Compression: Achieves an overall compression ratio of 3.1294 Bytes/Token with average fertility of 2.9073 Tokens/Word.
• GPT-4 Style Regex Pre-Tokenization: Isolates contractions ('s, 't, 're, 've), words, punctuation, and consecutive spaces.

📊 Benchmark Evaluation

📚 Training Corpus Composition

Trained on a 250 MB balanced multi-domain corpus:

• FineWeb (70% / 175 MB): High-quality filtered web prose
• WikiText-103 (10% / 25 MB): Encyclopedic, knowledge-dense prose
• CodeSearchNet (10% / 25 MB): Multi-language code and comments (Python, Java, Go, JS, PHP, Ruby)
• ArXiv / Math (5% / 12.5 MB): Scientific LaTeX and numerical notations
• OpenWebText (5% / 12.5 MB): Conversational and forum discourse

🚀 Quickstart Usage

You can load and use this tokenizer directly via Hugging Face transformers:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ZyroGod/exp-vocab-64k")

text = "def calculate_loss(predictions: torch.Tensor) -> float:\n    return float(loss.item())"
tokens = tokenizer.encode(text)
print("Encoded token IDs:", tokens)

decoded = tokenizer.decode(tokens)
print("Decoded text:", decoded)
assert decoded == text, "Roundtrip must be 100% lossless!"

📦 Artifacts Included

• tokenizer.json: Hugging Face Fast Tokenizer format
• vocab.json: Token-to-ID mapping
• merges.txt: Ranked BPE merge rules (GPT-2 compatible)
• tokenizer_config.json: Fast tokenizer configuration
• special_tokens_map.json: Reserved special tokens
• README.md: Auto-generated model card

---

Created with the Byte-Level BPE Tokenizer Engineering Framework.