Tokenizer-Ky-Ru

🚀 Fortaki Multi Mega BPE Tokenizer Suite (120K & 65K)

BPEF - is a family of ultra-optimized, custom Byte-Level BPE tokenizers available in two configurations: a massive flagship 120,000 token vocabulary and a lean, compute-efficient 65,000 token vocabulary.

Engineered from scratch by fortik11, this suite is specifically designed to eliminate the heavy "tokenization tax" on Cyrillic and highly agglutinative Central Asian languages, redefining compression standards for Russian and Kyrgyz NLP.

📊 Extreme Density Benchmarks (Stress Tests)

While global mainstream tokenizers (OpenAI GPT-4, Google Gemma, Meta Llama) yield a poor density of only 2.2 — 3.5 bytes/token on regional languages, the Fortaki suite achieves up to 3x higher efficiency, packing complex structures into record-low token footprints.

Breakthrough Performance Comparison:

Note: Even when halving the vocabulary from 120K to 65K, the Junior model retains over 88% of the flagship's compression efficiency due to a highly optimized BPE-merge heuristic strategy.

🧠 Core Architectural Advantages

1. 3x Context Window Expansion: By slashing the token count per sentence, your LLM will natively store up to 3 times more content in its Context Window compared to mainstream models. Say goodbye to early conversation amnesia in Russian and Kyrgyz.
2. Unprecedented TPS Boost (Inference Speed): LLM generation speed is strictly bound to token throughput. Because this tokenizer outputs complete complex stems or full grammatical morphemes in a single step ('пространство', 'человеческого'), your inference latency drops dramatically.
3. Flawless Morphological Awareness: The training dataset balancing avoids noisy byte-fallback splitting (\xd3) almost entirely. The vocabulary beautifully captures the inflected system of Russian and the deep agglutinative suffix chains of Kyrgyz ('сотвор' + 'ённое').
4. VRAM-Optimized Embedding Footprint: The 65K version offers a lean matrix, saving gigabytes of precious GPU memory for the actual model weights, while the 120K version offers absolute maximum semantic coverage for multi-turn complex reasoning tasks.

🛠️ Quick Start (Hugging Face Transformers)

You can call either version of the Fortaki tokenizer directly inside any modern pipeline:

from transformers import PreTrainedTokenizerFast

# Load your preferred version (120K or 65K)
tokenizer = PreTrainedTokenizerFast.from_pretrained("fortik11/<your-repository-name>")

text = "2026-жылы мамлекеттик кызматтарды санариптештирүү..."
encoded = tokenizer.encode(text)
print(f"Total tokens produced: {len(encoded)}")

📜 License

This suite is released under the Creative Commons Attribution 4.0 International (CC-BY-4.0) license. You are completely free to share, adapt, and use these tokenizers commercially, provided that appropriate credit is given to the original author: fortik11 (Fortaki).