indictok-acu

indictok‑acu — akshara‑constrained

A Unigram tokenizer for 22 Indian languages whose vocabulary is guaranteed almost never to split an akshara — the orthographic syllable a reader of a Brahmic script perceives as one character. A token that begins with a dependent vowel sign can only ever cut a consonant away from its own matra; this tokenizer removes that possibility rather than penalising it.

Full method and code: https://github.com/vivekvar-dl/indictok

The result

Like‑for‑like, 192k Unigram, same corpus, only the constraint differs:

98% of akshara‑splitting removed at effectively no compression cost at this scale. 16,184 vocabulary pieces pruned, round‑trip verified.

For comparison, akshara‑splitting in other published tokenizers: Sarvam‑30B 3.24%, MuRIL 5.30%, Llama‑4 76%.

Usage

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("vivekvar/indictok-acu")
tok.encode("भारत தமிழ் മലയാളം")

How the constraint works

The design turned on one fact: Unigram is the right substrate and BPE is not. A Unigram vocabulary is an unordered set of pieces scored by log‑probability, so deleting an akshara‑splitting piece simply removes it from the Viterbi search and the segmentation reroutes through legal pieces. BPE merges form an ordered chain — dropping a piece breaks every merge that depended on it.

Bare combining marks (a lone matra) are kept: they are the atomic fallback units that let arbitrary text be represented at all. Only multi‑character pieces beginning with a restricted mark are pruned.

The restricted set itself is not asserted — it is derived from 50M words of real text per script and ships with its evidence. One rule was rejected by that corpus (Malayalam 254‑ാം ordinal notation), and pieces beginning with that mark are correctly not pruned.

What we do not claim

Akshara‑splitting rate has no established link to downstream model quality. It measures orthographic correctness, nothing more. Given that fertility and morphological alignment both fail to predict downstream performance in the literature, the honest expectation is that this does too. It is published as a correctness property, not a performance claim.

License

Apache‑2.0.