kurdish-tokenizer-bpe-64k

Kurdish Tokenizer (BPE, 64k) — Kurmancî · Soranî · Zazakî

A byte-level BPE tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Why this exists

General-purpose multilingual tokenizers fragment Kurdish badly — especially Soranî, whose Arabic-script orthography differs enough from Arabic and Persian that shared subwords don't transfer well. Every extra token is extra compute and less usable context.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Against cl100k_base, this tokenizer needs 1.9× fewer tokens for the same Kurmancî text and 3.9× fewer for Soranî. It also beats every third-party tokenizer tested on all three dialects, including NLLB-200, which was purpose-built for 200 languages including Kurdish.

The Soranî gap is the widest because Arabic-script Kurdish is where general-purpose vocabularies fragment hardest — cl100k_base averages nearly 7 tokens per Soranî word.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-bpe-64k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî

<bos> / <eos> are appended automatically by the post-processor, so the output is training-ready without manual special-token bookkeeping.

Training data

Trained on 800k lines sampled from the training split of KurdishCorpus-clean, deliberately over-weighted toward Soranî and Zazakî relative to their share of the corpus, so the vocabulary isn't dominated by the far larger Kurmancî portion:

(For reference, Zazakî is only ~0.6% of the corpus by document count — it is up-weighted here, and its higher fertility below reflects how little Zazakî text exists to learn from, not a flaw in the sampling.)

Other variants

Four tokenizers were trained and benchmarked together; this repo is the BPE 64k one. BPE is the standard choice for causal language modeling (Llama/GPT/Mistral-style), and it has the lowest Kurmancî fertility of the four.

• kurdish-tokenizer-unigram-64k — best Soranî/Zazakî fertility of all four
• kurdish-tokenizer-bpe-32k — smaller vocab, smaller embedding table
• kurdish-tokenizer-unigram-32k — smaller vocab, smaller embedding table

Limitations

• Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation. The Zazakî figure rests on 166 documents and is the least robust of the three.
• Zazakî coverage is the weakest of the three, limited by how little Zazakî text exists.
• Lower fertility means fewer tokens for the same text; it does not by itself guarantee better downstream model quality.
• This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (BPE 64k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-bpe-64k}
}

---

Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.

Maintained by Alan Hesen.