pan-turkic-tokenizer

Pan-Turkic BPE Tokenizer

A SentencePiece BPE tokenizer with 65,536 vocabulary size, purpose-built for the Turkic language family. Covers Latin, Cyrillic, and Arabic scripts used across Turkic languages.

Overview

Most existing tokenizers fail on Turkic languages outside of Turkish — particularly on Cyrillic-script languages like Kazakh, Kyrgyz, Bashkir, and Tatar, where they fall back to byte-level tokenization. This tokenizer was trained specifically on a pan-Turkic corpus covering 20+ languages, and handles all major scripts natively.

Languages with strong coverage:

FLORES-200 Fertility Benchmark

Fertility = average tokens per word (lower is better). Evaluated on 1,012 sentences per language from the FLORES-200 devtest set.

Vocab sizes: Ours 65,536 · Kumru-2B 50,176 · GPT-2 50,257 · mT5 250,100 · NLLB-200 256,204 · XLM-R 250,002

Key result: Best on 7 of 10 Turkic languages. Achieves similar Turkic coverage to NLLB-200 (256K vocab) with a 4× smaller vocabulary.

For Cyrillic-script Turkic languages (Kazakh, Kyrgyz, Bashkir, Tatar), competing tokenizers degrade to byte-level encoding (10–11 tokens/word). This tokenizer maintains ~1.8 tokens/word on the same languages.

Notable Examples

Morphologically complex Turkish words encode efficiently:

"Cumhurbaşkanlığı"      → 1 token   # (Presidency)
"yapamayacaklarından"   → 3 tokens  # (from those they cannot do)
"sağlıklaştırılamayabileceklerden" → 6 tokens

Perfect round-trip for all supported scripts:

encode → decode  # lossless for Latin, Cyrillic, and Arabic Turkic scripts

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ArinUmut/pan-turkic-tokenizer")

# Turkish
tokenizer.encode("Türkiye Cumhuriyeti")

# Kazakh (Cyrillic)
tokenizer.encode("Алматы Қазақстанның ең үлкен қаласы")

# Uyghur (Arabic)
tokenizer.encode("بىز ئۇيغۇر تىلىدە سۆزلىشىمىز")

Specs

Limitations

• English fertility (2.27) is higher than English-specialized tokenizers, as the vocabulary is optimized for Turkic languages.