indictok-256k

indictok‑256k

A byte‑level‑free BPE tokenizer for 22 scheduled Indian languages across 12 scripts, 256,000 vocabulary. It beats Sarvam‑30B/105B — the open state of the art for Indic — by 14% on average fertility at comparable vocabulary size, and leads every tokenizer tested on worst‑case fertility, parity, and akshara‑splitting.

Full method, code, benchmark and 14 other variants: https://github.com/vivekvar-dl/indictok Results page: https://vivekvar-dl.github.io/indictok/

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("vivekvar/indictok-256k")
tok.encode("भारत একটি বিশাল দেশ தமிழ் மொழி മലയാളം ভাষা")

Or with the tokenizers library directly:

from tokenizers import Tokenizer
tok = Tokenizer.from_pretrained("vivekvar/indictok-256k")

Results

22 languages, 1,024 identical sentences from IN22‑Gen. Fertility is tokens per word, parity is Indic ÷ English cost, langs is how many of 22 the tokenizer actually encodes. Lower is better.

First on three measures: worst‑case fertility (3.23), parity (1.40× on the compression‑optimal variant), and akshara‑splitting (0.17%, 19× better than Sarvam‑30B).

What we do not claim

MuRIL and IndicBERT‑v2 post lower average fertility. We do not lead on that number. Both are encoder‑only models and both give something up for it:

• MuRIL drops ZERO WIDTH JOINER, which destroys Malayalam chillu forms (ള്‍ → ള്, a different letter). Only 39.1% of sentences survive with content intact, and it encodes 20 of 22 languages — Santali and Manipuri fail outright.
• IndicBERT‑v2 ships no decoder configuration. decode() returns WordPiece ## markers rather than text.

On MuRIL's own 14‑language set the gap narrows to 1.5% and this tokenizer leads on parity, but the headline average is theirs.

Every intrinsic metric here is a proxy. Ali et al. (2310.08754) trained 24 LLMs and found fertility does not reliably predict downstream performance. We make no claim that any number here predicts model quality.

How it was built

• Character‑level, not byte‑level. Indic codepoints are 3‑byte UTF‑8; byte‑level BPE shreds a single character into fragments. Llama‑3 and Qwen contain zero readable Devanagari characters in their vocabularies.
• Script rules validated against 50M words per script, not asserted. 137 accepted, 1 rejected — Malayalam 254‑ാം ordinal notation, which a hand‑written rule would have wrongly flagged.
• Byte fallback with the byte tokens present, so no character is ever silently deleted. Round‑trip is verified against scripts the tokenizer was never trained on.
• Training data is CC‑0 (Sangraha), so the output is unencumbered.

Variants

Fifteen tokenizers in total, on GitHub — including a 128k that leads on worst‑case and parity, per‑language Hindi tokenizers, and the akshara‑constrained variant (vivekvar/indictok‑acu) that removes 97% of akshara‑splitting for a 3.2% compression cost.

License

Apache‑2.0.