uzbek-gpt-103m

uzbek-gpt-103m

A 103M-parameter, decoder-only transformer for Uzbek, pretrained from scratch — not fine-tuned from a multilingual model. Trained on one consumer GPU in about 3.4 hours for about $3.60, it predicts held-out Uzbek text better than mGPT-1.3B (13× larger, used as-is or fine-tuned on Uzbek) and than current Qwen models of up to 1.7B parameters.

The paired tokenizer is IslombekT/uzbek-bpe-16k.

• Author: Islombek Turdiyev (GitHub · website)
• Language: Uzbek (Latin script, uzn_Latn)
• Architecture: decoder-only transformer with RoPE, RMSNorm, and SwiGLU (no output projection after the attention heads)
• Parameters: ~103M
• Tokenizer: uzbek-bpe-16k (16,384 tokens)
• Paper and code: github.com/TurdiyevIslombek/uzbek-gpt-from-scratch

Results

Models are compared by bits-per-byte: how many bits a model needs per byte of text. Unlike perplexity it doesn't depend on the tokenizer, so it's fair across models that tokenize differently. Lower is better.

The test set is 474 Uzbek documents from the validation split that share no 24-token passage with the training data, in their original web spelling. Every model scores the same ~800-byte spans with the same preceding context.

Paired bootstrap over the 474 test documents, 10,000 resamples. Two smaller Qwen models (Qwen3-0.6B, Qwen2.5-0.5B) score 1.905 and 2.357. The Qwen models were not fine-tuned on Uzbek and may have seen some of these pages in training; both would favour them.

What the advantage comes from

To separate the tokenizer from the training data, the same architecture was trained a second time on exactly the same documents, with mGPT's 100,000-token tokenizer instead of uzbek-bpe-16k.

Difference (ablation minus this model), 95% CI: web −0.007 [−0.009, −0.005]; normalized +0.005 [+0.002, +0.008].

• For quality, the tokenizer doesn't matter much. The two models reach comparable quality, within 0.007 bits/byte of each other, and each is better on the spelling it trained on (this model's tokenizer normalizes apostrophes; mGPT's doesn't).
• The advantage over mGPT comes from training on Uzbek text. The ablation uses mGPT's own tokenizer and is 0.046 ahead of fine-tuned mGPT.
• The tokenizer's payoff is efficiency. Comparable quality with less than half the parameters and about a third of the training compute, and 1.749 tokens per word — about half of GPT-2 and 45% fewer than mGPT's tokenizer.

The ablation matched training data, not compute (which favours it), and each model was trained once.

Does fine-tuning mGPT catch up?

It keeps improving but is still 0.026 behind at 10M tokens. A rough extrapolation from two points suggests catching up could take on the order of 1 billion tokens — about as much Uzbek text as this model was trained on. mGPT's tokenizer isn't what holds it back: the ablation shows the same tokenizer reaches 1.031 with a billion Uzbek tokens.

Sample output

One continuation per prompt, fixed seed, not selected (first sentence):

Grammatical, news- and law-style Uzbek — but a small base model has no understanding behind the text. Longer outputs invent names and products, and one of the four samples in the paper doesn't make sense.

Evaluation correction (September 2026)

An earlier version of this card reported 1.105 bits/byte and claimed the advantage came entirely from the tokenizer. An audit found four problems, each flattering this model or its tokenizer: the test set was the opening of the training split; the validation split was about 39% duplicated from training data; the test text was stored in this tokenizer's normalized spelling, which the other models hadn't learned; and the first tokenizer ablation had read the whole corpus (including the test documents' region) and stopped halfway, so it scored 1.048 and made the tokenizer look responsible for about a quarter of the advantage. The ablation was retrained on exactly this model's training documents. All results above are re-measured on a deduplicated test set in its original web spelling. Running the corrected code on the old test set gives 1.1078 / 1.1653 / 1.1604, within 0.003 of the old numbers (1.1050 / 1.1628 / 1.1579); the old code divided by the byte count of the whole file instead of the bytes it scored. Scripts are in the repository.

Training

* Measured on the validation split, later found to be about 39% duplicated from training data, so it's optimistic. Use the bits-per-byte results for a clean measure.

Intended use

Research and experimentation with Uzbek language modeling, tokenizer studies, and low-resource-NLP baselines. This is a base model: it predicts the next token and has not been instruction-tuned or aligned, so it is not a chat assistant and should not be deployed in user-facing products without further training and evaluation.

Limitations and bias

Trained on web text, so it reflects the coverage, style, and biases of that data and can produce inaccurate or inappropriate text. Latin-script Uzbek only. As a small base model it invents facts and has no knowledge beyond its training data. Evaluation is intrinsic only (bits-per-byte); no downstream tasks were tested. One training run.

How to use

The model uses a custom architecture, so it doesn't load with AutoModel. The definition (model.py, also in this repo with config.json), a generation script, and all evaluation code are in the repository:

git clone https://github.com/TurdiyevIslombek/uzbek-gpt-from-scratch
cd uzbek-gpt-from-scratch
pip install -r requirements.txt          # needs transformers 5 or newer for the tokenizer
curl -LO https://huggingface.co/IslombekT/uzbek-gpt-103m/resolve/main/model.safetensors
python generate.py --ckpt model.safetensors --prompt "Oʻzbekiston "

github.com/TurdiyevIslombek/uzbek-gpt-from-scratch

Acknowledgements

Trained on FineWeb-2 (uzn_Latn, ODC-By 1.0) from Hugging Face. I used Claude (Anthropic) to learn the methods, write and debug code, and edit writing. It also flagged two of the four problems in the evaluation audit (the test-set overlap and the mismatch in the first ablation run), which I confirmed. I ran every check, trained every model and ran every experiment myself.

Citation

@misc{turdiyev2026uzbekgpt103m,
  title  = {uzbek-gpt-103m: a from-scratch language model for Uzbek},
  author = {Turdiyev, Islombek},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/IslombekT/uzbek-gpt-103m}}
}

Related

• Tokenizer: IslombekT/uzbek-bpe-16k
• Code, paper, and study: uzbek-gpt-from-scratch
• More of my work: turdiyevislombek.github.io · EduBoost