otk-bpe

OTK-BPE: Optimized BPE Tokenizers

This repository hosts the OTK-BPE family of production-grade Byte-Level BPE (BBPE) tokenizers for regional and multilingual language models. It now covers Swahili, Kinyarwanda, and a merged French + Kinyarwanda + English + Swahili vocabulary — each available at three vocab sizes: 50k, 100k, and 150k — so you can pick the tradeoff between compactness and coverage that fits your use case.

(Companion repo: olaverse/otk-bpe-50k covers the Nigerian-languages family — Yoruba, Igbo, Hausa, Pidgin, and a unified Naija tokenizer, all fixed at 50k.)

Which one should I use?

150k is the recommended default across all three families — in every benchmark below, fertility and entity-handling both improved monotonically from 50k → 100k → 150k, with no exceptions. There's no size in this range where a smaller vocab wins outright.

Key Features

• Byte-Level BPE (BBPE): maps raw UTF-8 bytes to printable characters — 0.00% Out-Of-Vocabulary, zero [UNK] tokens, by construction.
• Diacritic Preservation & Normalization: NFC normalization inside the pre-tokenization chain, so accents and combining diacritics don't get split into decomposed code points.
• Code-Mixed English Support: the Swahili and Kinyarwanda tokenizers each blend a English-Wikipedia component into training; the merged tokenizer treats English as a full first-class language rather than a minority blend.
• Emoji & Symbol Merging: curated emoji vocab is injected during training so common emoji merge into single tokens instead of fragmenting.

Tokenizer Models in This Repository

All loaded from olaverse/otk-bpe, selected via the subfolder argument.

Performance Benchmarks

Fertility = average tokens per word (lower is better). Entity fragmentation = share of capitalized/likely-proper-noun words split into more than one token (lower is better) — this metric predicts downstream NER difficulty before any model is even trained.

Swahili benchmark uses the MasakhaNEWS Swahili test split (42,494 entity-candidate words checked) — a real, curated held-out set, never seen during training.

Kinyarwanda, French, and English benchmarks use a held-out, non-training slice of the same streaming sources (FineWeb-2 / Wikipedia) rather than a curated test set — MasakhaNEWS does not include Kinyarwanda (only the related-but-distinct Rundi/Kirundi), so no equivalent curated benchmark exists for that language yet. Treat these three languages' numbers as directionally reliable, not as rigorously verified as the Swahili figures.

Swahili

Kinyarwanda

Kinyarwanda is where general-purpose multilingual tokenizers are weakest across the board — even the smallest dedicated tokenizer here (kin-50k) beats every baseline on fertility, and kin-150k beats every baseline on both metrics simultaneously.

French

English

English is the merged tokenizer's weakest relative showing — expected, since none of the 4 languages get a dedicated vocabulary in the merged design, and mBERT/AfroXLMR both have far larger, English-rich vocabularies to draw on. merged-150k still lands within ~1% of mBERT's fertility despite covering 3 additional languages in the same vocabulary.

Reading the entity fragmentation numbers honestly

At 150k vocab, OTK-BPE tokenizers win entity fragmentation outright, not just fertility, for Swahili and Kinyarwanda — sw-150k and kin-150k both have the lowest fragmentation rate of every tokenizer tested, dedicated or general-purpose. The vocab-size increase from 50k to 150k didn't just narrow the gap to larger multilingual tokenizers on this metric, it closed and then reversed it for these two languages specifically.

The exceptions are French and English, where mBERT — not AfroXLMR — has the best entity handling (0.388 and 0.285 respectively), edging out even the merged tokenizer. This makes sense: mBERT's vocabulary, while smaller than AfroXLMR's, was built with heavy exposure to major European-language Wikipedia text including large volumes of proper nouns, and English/French are exactly where that shows. The merged tokenizer's own English and French entity handling (0.558 and 0.428) is competitive but not best-in-class for those two languages specifically — a real, honest tradeoff of covering 4 languages in one shared vocabulary rather than dedicating full budget to either.

Net read: don't assume general-purpose tokenizers automatically win on entity handling just because they have larger vocabularies — that held at the smaller 50k vocab size tested earlier in this project, but stopped holding once vocab size reached 150k for the languages that needed it most.

How to Use

Installation

pip install tokenizers transformers huggingface_hub

Method A: Standard Transformers Loading

from transformers import AutoTokenizer

# Load the Swahili tokenizer at your chosen size
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="sw-150k")

text = "Habari yako? Leo ni siku nzuri sana 😊"
inputs = tokenizer(text)

print("Tokens:", tokenizer.tokenize(text))
print("IDs:", inputs["input_ids"])
print("Decoded:", tokenizer.decode(inputs["input_ids"]))

# Kinyarwanda
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="kin-150k")

# Merged (French + Kinyarwanda + English + Swahili)
tokenizer = AutoTokenizer.from_pretrained("olaverse/otk-bpe", subfolder="merged-150k")

Method B: Lightweight Raw BPE Loading

from olaverse import Tokenizer

# Supports: "sw-50k", "sw-100k", "sw-150k", "kin-50k", "kin-100k", "kin-150k",
#           "merged-50k", "merged-100k", "merged-150k"
tokenizer = Tokenizer("sw-150k")

ids = tokenizer.encode("Habari yako? Leo ni siku nzuri sana 😊")
print("IDs:", ids)
print("Decoded:", tokenizer.decode(ids))

Datasets Used for Training

MasakhaNEWS Swahili test split was excluded from all training corpora and reserved exclusively for the fertility/entity benchmarks above.

Links

• Companion repo: Nigerian languages (otk-bpe-50k)
• Olaverse Library Docs

License

Apache-2.0. Training data licenses noted per source above; please retain attribution to upstream datasets.