uzbek-bpe-16k

uzbek-bpe-16k

A byte-level BPE tokenizer built for Uzbek (Latin script). Multilingual tokenizers split Uzbek into too many pieces and treat the several ways of typing the apostrophe in oʻ and gʻ as unrelated characters. This tokenizer is trained on Uzbek and normalizes most of those variants, so it uses fewer tokens per word than any of the ten widely used tokenizers it was compared with, including ones with vocabularies 6 to 15 times larger.

• Author: Islombek Turdiyev (GitHub · website)
• Language: Uzbek (Latin script, uzn_Latn)
• Algorithm: byte-level BPE with GPT-2-style pre-tokenization
• Vocabulary size: 16,384, including one special token, <|endoftext|>
• Fertility: 1.749 tokens per word · 4.641 bytes per token (474 held-out documents, web spelling)

Why it exists

Multilingual tokenizers share one vocabulary across dozens of languages, so a low-resource language gets few merges and breaks into many small pieces. That lengthens every sequence, which costs money, time, and context. A tokenizer trained on Uzbek learns merges that follow Uzbek word structure, including its stacked suffixes (kitob → kitoblar → kitoblarimiz).

Byte-level means nothing is ever out of vocabulary: any text, including Cyrillic or emoji, can still be encoded, one byte at a time if necessary.

Apostrophe normalization

The mark in oʻ and gʻ is typed many ways on the web: ', ‘, ’, `, and the official ʻ (U+02BB). Before BPE, this tokenizer applies these rules, in order:

1. ’ (U+2019) and ` become '.
2. o', O', g', G' become oʻ, Oʻ, gʻ, Gʻ.
3. An apostrophe between two letters becomes the tutuq belgisi ʼ (U+02BC), as in maʼlumot.

So o'zbek, o’zbek, o`zbek and oʻzbek all produce the same tokens.

Known gaps.

• The left quote ‘ (U+2018) is not normalized. In a sample of 3,000 raw web documents it is the most common way people type the mark (34%, against 23% for the official ʻ), so o‘zbek costs one extra token. In the 474 test documents it is 19% of the marks, and normalizing it too would cut the token count there by 2.3% (1.749 → 1.709 tokens per word). It is left unchanged here because the paired model was trained with the tokenizer as it is.
• Rule 2 has no word-boundary check, so a closing quote after a word ending in o also changes: 'dunyo' becomes 'dunyoʻ.

The normalization is one-way: the original apostrophe character can't be recovered after decoding.

Efficiency

Tokens per word (fertility, lower is better) and bytes per token (higher is better), on 474 held-out Uzbek documents (200,818 words) in their original web spelling — text this tokenizer never saw in training:

About half the tokens of GPT-2 and GPT-3.5/4, 45% fewer than mGPT, 21% fewer than XLM-R. An earlier measurement on a different sample gave 1.839, with the same ranking.

Does the tokenizer affect model quality?

Only slightly, and not in one consistent direction. What it mainly affects is size and cost. The same 12-layer transformer was trained twice on exactly the same Uzbek documents (two passes each): once with this tokenizer, once with mGPT's 100,000-token tokenizer.

On held-out text in its original web spelling the mGPT-tokenizer model is 0.007 bits/byte better (95% CI [0.005, 0.009], paired bootstrap over documents); on the same text in this tokenizer's normalized spelling, the uzbek-bpe-16k model is 0.005 better ([0.002, 0.008]). Each wins on the spelling it was trained on. So this tokenizer doesn't make a model predict Uzbek better; it reaches comparable quality with less than half the parameters and about a third of the training compute, and every text costs fewer tokens afterwards (efficiency table above).

An earlier version of this card reported a 0.010 bits/byte quality gain from the tokenizer. That came from a flawed first ablation run (it read the whole corpus and stopped halfway); the retrained ablation above replaces it.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("IslombekT/uzbek-bpe-16k")

ids = tok("Oʻzbekiston Respublikasi poytaxti Toshkent shahri.")["input_ids"]
print(len(ids))
print(tok.decode(ids))

This needs transformers 5 or newer. With an older version, load the same file directly:

from huggingface_hub import hf_hub_download
from transformers import PreTrainedTokenizerFast

tok = PreTrainedTokenizerFast(
    tokenizer_file=hf_hub_download("IslombekT/uzbek-bpe-16k", "tokenizer.json"),
    eos_token="<|endoftext|>",
)

Files

• tokenizer.json — the tokenizer: the apostrophe normalization above, then byte-level BPE. Trained with Hugging Face's tokenizers library.
• uz_tokenizer.py — my from-scratch Python implementation of the same algorithm and normalization rules, written to learn how BPE works. Pure Python is too slow at corpus scale, so the released tokenizer.json was trained with the library.

Intended use

Tokenizing Uzbek (Latin-script) text for language modeling, fine-tuning, and NLP experiments; training Uzbek models; and as a baseline for tokenizer studies on low-resource languages.

Limitations

• Latin script only. Cyrillic Uzbek still encodes (byte-level) but inefficiently.
• Vocabulary size was not tuned. 16,384 reaches comparable quality to 100,000 with a much smaller model; it is not shown to be optimal.
• The normalization's effect was not measured separately, and it misses the left quote ‘ (see above).
• Alphabet reform. On text in the amended alphabet (ş ç ö ğ), fertility rises 38%; converting those letters back to sh ch oʻ gʻ before tokenizing restores it exactly. See the repository's alphabet study.
• Trained on web text (FineWeb-2), so merges reflect the vocabulary and style of the Uzbek web.

Paired model

Used to train IslombekT/uzbek-gpt-103m: 1.038 bits/byte on the held-out set, ahead of mGPT-1.3B zero-shot (1.080) and fine-tuned (1.077), and of current Qwen models up to 1.7B parameters.

An earlier version of this card quoted 1.105 bits/byte and described the tokenizer as the single highest-leverage decision. That figure came from a test set later found to overlap the training data; the model card explains the correction.

Acknowledgements

Trained on FineWeb-2 (uzn_Latn, ODC-By 1.0) from Hugging Face. I used Claude (Anthropic) to learn the methods, write and debug code, and edit writing. It also flagged two of the four problems in the evaluation audit (the test-set overlap and the mismatch in the first ablation run), which I confirmed. I built the tokenizer and ran every measurement myself.

Citation

@misc{turdiyev2026uzbekbpe16k,
  title  = {uzbek-bpe-16k: a byte-pair-encoding tokenizer for Uzbek},
  author = {Turdiyev, Islombek},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/IslombekT/uzbek-bpe-16k}}
}

Related

• Model: IslombekT/uzbek-gpt-103m
• Code, paper, and study: uzbek-gpt-from-scratch
• More of my work: turdiyevislombek.github.io · EduBoost