akshara-tokenizer

IYRA Akshara Tokenizer (IAT)

Linguistically-correct tokenization for six Brahmic (Indic) scripts. The tokenizer treats the Akshara (orthographic syllable) as the atomic writing unit rather than the byte. It has two layers: a rule-based Akshara segmenter (pure Unicode rules, no model) that never splits an Akshara, and a SentencePiece model trained on the segmented stream. The Python library that provides the segmenter and the high-level tokenizer lives on GitHub and PyPI, linked below.

This repository holds two model versions:

Supported scripts

What is new in v1.2: lossless round-trip

In v1.2 each Akshara the map covers is replaced by a single private-use codepoint before SentencePiece, so no token boundary can fall inside a covered Akshara. decode(encode(text)) is byte-identical, reconstructed from the id stream alone with no side channel. This was verified on a 20-string probe set covering the six scripts, mixed script, nukta forms, conjuncts, and whitespace edges (20 of 20), and on 1,750 FLORES-200 devtest lines, 250 each from the six Indic scripts and English (1,750 of 1,750). v1.1 could not round-trip at all, because its pipeline space-joins aksharas and the model normalizes whitespace. 45,274 of the 64,000 v1.2 vocabulary pieces span two or more aksharas.

Fertility

FLORES-200 devtest, tokens per whitespace-delimited word (lower is better). Numbers match the GitHub README (benchmark_2026_07/results_v1_2.md).

v1.2 roughly halves v1.1's token count on every Indic script and stays far below Qwen3-14B. On English, Qwen3 is more efficient, as expected for an English-centric byte-level BPE.

Against Indic-specific tokenizers

Qwen3-14B is a general multilingual BPE and a weak baseline for an Indic claim. Against tokenizers built for Indian languages, on the same FLORES-200 devtest: budget-matched against sarvam-1 (68,096 vocab, closest to v1.2's 64,000), v1.2 uses fewer tokens on all six native scripts, by 2.00 to 14.15 percent. Against sarvam-30b (262,144 vocab, 4.1x the budget) it wins five of six, losing Bengali by 6.76 percent.

That comparison is not like for like. sarvam and Krutrim are full LLM tokenizers carrying English, code and multilingual coverage in one vocabulary, while v1.2 spends 92.25 percent of its 64,000 slots on Indic aksharas. A native-script win partly reflects that specialisation, not engineering alone. The classification rule behind that 92.25 percent is stated in full under "Vocabulary classifier" in results_competitor_comparison.md, so the figure is checkable rather than asserted. An earlier revision of this card said 91.94 percent; that figure could not be reproduced and is superseded.

Round-trip differs too. v1.2 is 1,012 of 1,012 on all six scripts; Qwen3-14B ranges from 403 to 1,011. That is not data loss: Qwen3 applies Unicode NFC normalization inside its tokenizer, so its output is canonically equivalent to the input and renders identically without being byte-identical. It is a reasonable design choice. The tradeoff is canonicalization against exact reconstruction.

Where this tokenizer is the wrong choice

On Latin-script input the advantage inverts, by design rather than by defect: with 92.25 percent of the vocabulary spent on Indic aksharas, only 2,251 slots (3.52 percent) go to Latin pieces, at a mean length of 3.44 characters. On Dakshina human romanizations v1.2 uses more tokens than Qwen3-14B, by 23.4 percent on romanized Hindi, 16.0 on Punjabi and 12.8 on Tamil. On code-switched Hinglish it is last of five at 2.377 tokens per word, against 1.448 for sarvam-30b at 262,144 vocab.

If your workload is predominantly romanized or code-switched Latin text, a large general-purpose tokenizer will serve you better. If it is native script, this one will not.

Full tables and method in benchmark_2026_07/results_competitor_comparison.md on GitHub. That file also reports tokens per 100 characters alongside tokens per word, for cross-script comparison. It preserves the tokenizer-to-tokenizer ratios within each script, so it changes no conclusion above.

Akshara split rate

FLORES-200 devtest, percent of aksharas whose token boundaries fall inside them (lower is better).

For v1.2, a covered akshara is one codepoint, so the residual is only the coverage-trim tail, not vocabulary pressure. v1.2 byte-fallback is near zero on all six scripts (0.00 to 0.04 percent).

The mapping table is required

The v1.2 ids are not decodable without the mapping table. Each piece is built from private-use codepoints, and the map (akshara_tokenizer_v1_2.map.json) translates those back to aksharas. The map ships next to the model and is bound to it by sha256, so loading refuses a mismatched pair. Always fetch the model and its map together, and do not store ids without both.

Versions and compatibility

v1.1 and v1.2 produce different, incompatible id streams. Nothing in the API records which model produced a stream, so decoding v1.1 ids with the v1.2 model returns silent garbage rather than an error. If you already have a corpus tokenized with v1.1, keep using v1.1 (the files remain in this repository, and PyPI 1.1.0 installs it) rather than upgrading in place.

Precomposed nukta letters (Gurmukhi and Bengali)

Precomposed nukta letters encode to 2 or 3 pieces rather than 1, because the map holds their decomposed equivalents: Gurmukhi U+0A36 as U+0A38 followed by U+0A3C, Bengali U+09DF as U+09AF followed by U+09BC. This costs tokens, never correctness: these aksharas carry zero byte-fallback and round-trip remains byte-identical. On FLORES-200 it affects 0.33 percent of Gurmukhi aksharas.

Gurmukhi and Bengali pay the same penalty, and NFC-normalizing the input recovers it:

Round-trip is unaffected by normalization form: 1,012 of 1,012 lines in all eighteen cells, six scripts across as-is, NFC and NFD input.

NFD input costs tokens instead, up to 33.61 percent on Kannada, but sarvam-1 degrades 37.91 percent on the same input, so that is a general Brahmic-NFD effect rather than something specific to this tokenizer.

The NFC recommendation is a measured conclusion, not an untested workaround. Rebalancing the training corpus toward the precomposed forms was tested at three levels, without touching the map. It recovers at most 41.7 percent of the Gurmukhi penalty and 17.3 percent of the Bengali penalty, while degrading other scripts, because within a fixed corpus every occurrence one form gains the other loses. The penalty is therefore structural within a fixed vocabulary budget, and it is the price of byte-exact round-trip. Round-trip held at 1,012 of 1,012 in all eighteen cells throughout.

Detail in benchmark_2026_07/results_bengali_diagnostic.md and benchmark_2026_07/results_arm_n_normalization.md on GitHub.

Usage (v1.2, fetching files from this repo)

Install the library (it provides the segmenter and the AksharaTokenizer class), then download both v1.2 files and load them together:

pip install "akshara-tokenizer[model]"

from huggingface_hub import hf_hub_download
from akshara_tokenizer import AksharaTokenizer

repo = "GursimranSinghBasra/akshara-tokenizer"
model_path = hf_hub_download(repo, "akshara_tokenizer_v1_2.model")
map_path = hf_hub_download(repo, "akshara_tokenizer_v1_2.map.json")

tok = AksharaTokenizer.from_files(model_path, map_path)

tok.encode("न्याय दर्शन")            # [7645, 531, 7572]
tok.decode([7645, 531, 7572])        # "न्याय दर्शन"   (byte-identical)
tok.pieces("ਪੰਜਾਬ")                  # ["ਪੰਜਾਬ"]

The simpler path is PyPI: pip install "akshara-tokenizer[model]" already bundles the v1.2 model and map, so AksharaTokenizer.load() works with no download.

The rule-based segmenter alone needs no model:

from akshara_tokenizer import segment_aksharas, count_aksharas

segment_aksharas("ਨਿਆਯ ਦਰਸ਼ਨ")
# ['ਨਿ', 'ਆ', 'ਯ', ' ', 'ਦ', 'ਰ', 'ਸ਼', 'ਨ']

count_aksharas("ਪੰਜਾਬ")
# 3

Usage (v1.1, legacy)

v1.1 has no map and does not round-trip. Fetch the model and use the space-join pipeline:

from huggingface_hub import hf_hub_download
import sentencepiece as spm
from akshara_tokenizer import segment_aksharas

model_path = hf_hub_download("GursimranSinghBasra/akshara-tokenizer", "akshara_tokenizer_v1_1.model")
sp = spm.SentencePieceProcessor(model_file=model_path)
ids = sp.encode(" ".join(segment_aksharas("न्याय दर्शन")))

Links

• Library, PyPI package, and reproducible benchmarks: https://github.com/1322Guru/akshara-tokenizer
• PyPI: https://pypi.org/project/akshara-tokenizer/ (pip install "akshara-tokenizer[model]")
• Fertility and split-rate methodology and raw numbers: benchmark_2026_07/results_v1_2.md in the GitHub repository.
• Competitor comparison against sarvam-1, sarvam-30b and Krutrim-2: benchmark_2026_07/results_competitor_comparison.md
• Normalization robustness, as-is against NFC against NFD: benchmark_2026_07/results_bengali_diagnostic.md
• Qwen3 round-trip verified against stock Qwen/Qwen3-14B: benchmark_2026_07/results_qwen_roundtrip_check.md
• Romanized Indic on Dakshina: benchmark_2026_07/results_dakshina_romanized.md
• Whether corpus rebalancing can close the precomposed nukta penalty: benchmark_2026_07/results_arm_n_normalization.md

Patent and copyright

Patent pending: Indian provisional patent application 202611071450. Copyright 2026 Gursimran Singh.

Related projects

• Live demo of this tokenizer: https://akshara.iyraproject.com
• IYRA: https://iyraproject.com
• NyayaBench: https://nyayabench.com

IYRA is a separate project by the same author. IYRA does not use this tokenizer: it runs on the Qwen3-14B tokenizer. AksharaTokenizer is developed independently and shares no runtime code with IYRA. NyayaBench is the benchmark used to evaluate IYRA and is likewise unrelated to this tokenizer.

License

Apache License 2.0. See LICENSE and NOTICE.