Downloads · 30 days
0
Abzalbek89/kk-tokenizer-sp-unigram-32k
kk-tokenizer-sp-unigram-32k is a machine learning model from Abzalbek89. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
SentencePiece Unigram tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000. Trained with sentencepiece on the cleaned Kazakh corpus Abzalbek89/corpusclean (~1.5M documents).
Downloads · 30 days
0
Access
Public
Updated Apr 30, 2026
Repo size
974 KB
Likes
0
Public
Click a slice to open those files.
.model974 KB · 56%
From the Hugging Face model README
SentencePiece Unigram tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000.
Trained with sentencepiece on the cleaned Kazakh corpus
Abzalbek89/corpus_clean (~1.5M documents).
SentencePiece Unigram tokenizer trained on Kazakh with full character coverage (character_coverage=1.0) and NFKC normalization.
| Metric | Value |
|---|---|
| Rank | 4 / 13 (lower fertility is better) |
| Vocabulary size | 32,000 |
| Fertility (tokens / word) | 1.762 |
| Compression (chars / token) | 4.454 |
| Compression (bytes / token) | 8.065 |
| Total tokens / words | 5,225,471 / 2,966,207 |
Evaluated on 2,966,207 whitespace-words from the validation split of Abzalbek89/corpus_clean (14,831+ documents, ~3M words). Full ranking and methodology: Abzalbek89/kk-tokenizer-fertility-baseline.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-sp-unigram-32k")
ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
add_special_tokens=False)
print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))
Abzalbek89/corpus_cleanAbzalbek89/kk-tokenizer-fertility-baseline@misc{kk_tokenizer_2026,
title = {Tokenizer Optimization for Kazakh Small Language Models},
author = {Abzalbek Ulasbek},
year = {2026},
howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-sp-unigram-32k}},
}
Apache 2.0