kk-tokenizer-morph-hf-bpe-32k

kk-tokenizer-morph-hf-bpe-32k

Byte-Level BPE tokenizer for the Kazakh language (Cyrillic), vocabulary size 32,000. Trained with huggingface/tokenizers on the cleaned Kazakh corpus Abzalbek89/corpus_clean (~1.5M documents).

Morphology-aware BPE tokenizer. Training corpus was first segmented by an unsupervised Morfessor model trained on top-1M Kazakh word forms; BPE then merges only within morpheme boundaries. At inference, apply the same Morfessor segmentation before encoding.

Held-out fertility result

Evaluated on 2,966,207 whitespace-words from the validation split of Abzalbek89/corpus_clean (14,831+ documents, ~3M words). Full ranking and methodology: Abzalbek89/kk-tokenizer-fertility-baseline.

Morphology-aware inference

This tokenizer expects input that has been pre-segmented by the Morfessor model shipped in Abzalbek89/kk-tokenizer-fertility-baseline. Without that step the tokenizer will still encode the input but the morpheme-aware merge bias is lost.

import morfessor
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('Abzalbek89/kk-tokenizer-morph-hf-bpe-32k')
morf_path = hf_hub_download(
    'Abzalbek89/kk-tokenizer-fertility-baseline', 'morfessor.bin', repo_type='dataset')
io = morfessor.MorfessorIO()
model = io.read_binary_model_file(morf_path)

def morph_segment(text):
    out = []
    for w in text.lower().split():
        morphemes, _ = model.viterbi_segment(w)
        out.extend(morphemes)
    return ' '.join(out)

ids = tok.encode(morph_segment('Қазақстан Республикасы'), add_special_tokens=False)

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Abzalbek89/kk-tokenizer-morph-hf-bpe-32k")
ids = tok.encode("Қазақстан Республикасының мемлекеттік тілі қазақ тілі.",
                 add_special_tokens=False)
print(len(ids), tok.tokenize("Қазақстан Республикасының мемлекеттік тілі қазақ тілі."))

Companion artifacts

• Training corpus: Abzalbek89/corpus_clean
• Fertility benchmark + reproducible scripts: Abzalbek89/kk-tokenizer-fertility-baseline
• Sibling tokenizers (BPE / Unigram / SP / morph-aware): see the dataset README for the full table.

Citation

@misc{kk_tokenizer_2026,
  title  = {Tokenizer Optimization for Kazakh Small Language Models},
  author = {Abzalbek Ulasbek},
  year   = {2026},
  howpublished = {Hugging Face Hub: \url{https://huggingface.co/Abzalbek89/kk-tokenizer-morph-hf-bpe-32k}},
}

License

Apache 2.0