bornomala-bengali-tokenizer

BMBT: Bornomala's Bengali Tokenizer

A Bengali-first tokenizer that parses Bengali's own akshara grammar with a hand-written finite-state parser, instead of discovering structure statistically, and never splits a conjunct. It is Project Bornomala's flagship tokenizer, and the second, independently trained component of Project Bornomala, a non-commercial research effort from West Bengal to build a Bengali-first, dialect-aware language model and to preserve the Bengali language and its dialects.

v1 (bmbt-64k). Trained on the same literary-weighted corpus as this project's earlier BPE-over-grapheme-clusters model (Wikisource, AI4Bharat Sangraha, Wikipedia, XL-Sum news), 64k vocabulary. The simpler sibling model, bn-bpe-64k, has its own listing at konko/bengali-bpe-tokenizer.

What makes BMBT different from a grapheme-cluster BPE tokenizer

• Grammar, not statistics. bn-bpe-64k finds its atomic unit (the grapheme cluster) with Unicode's generic UAX #29 algorithm, then lets BPE merges discover everything else statistically. BMBT instead parses Bengali's own generative virama grammar with a hand-written finite-state parser (akshara.py) before any merge is learned, so every atom is a real akshara: a whole onset-conjunct-plus-vowel unit, not just "whatever UAX #29 groups together."
• Featural decomposition as a first-class output. BMBT's featurize() returns each akshara's real structural decomposition (onset consonants, which carry a nukta, the vowel, trailing modifiers, whether a ZWJ/ZWNJ occurred) directly from the tokenizer, not as an embedding-layer afterthought bolted on later.
• Conjunct integrity by construction, not by vocabulary luck. A grammar-level guarantee: a virama-joined consonant conjunct cannot be split across a token boundary, because the parser never produces an atom that cuts through one.
• Provably tied, not beaten, on raw efficiency, and that is the honest finding. A constrained BPE (never split an akshara) cannot beat an unconstrained one on raw token count; Bengali's own akshara-grammar boundaries turn out to be nearly isomorphic to grapheme-cluster boundaries on well-formed text. Measured against bn-bpe-64k on six disjoint held-out registers, the two tie exactly on five and BMBT edges ahead by a measured 0.001 fertility on the sixth (FLORES+). Reported exactly as measured, not rounded away in either direction.
• A morphology-aware variant exists (bmbt-64k-morph, not the artifact in this repository): an opt-in layer that aligns token boundaries to Bengali's own suffix structure, reaching 100% of reachable morpheme seams with zero conjunct-integrity violations, at a real fertility cost (21.7%-42.1% depending on register). See the GitHub repository's docs/bmbt-morphology.md.

Performance

Measured on 828 held-out Bengali Wikipedia lines (unseen during training), against real public tokenizers from 18 other systems on identical NFC-normalised text. Four further disjoint held-out registers (literary/ formal, general web, news, FLORES+) confirm the same ranking; a sixth (Banglish, romanized Bengali) is reported separately below because it is deliberately outside this tokenizer's design scope. Full six-register tables: repository's benchmarks/bengali-comparison.md.

Lower fertility and lower destructive rate are better; higher STRR and bytes per token are better. Destructive rate counts only splits that sever something real (a stranded virama, a detached nukta), not a harmless consonant-cluster/vowel-sign seam. Every general tokenizer measured breaks between 0.9% and 15.5% of Bengali conjuncts destructively on this held-out set; this one breaks 0.04%.

On the FLORES+ register (2,009 professionally translated sentences, the same corpus an external tokenizer-fertility paper's own published Bengali numbers were built from), BMBT measures fertility 1.240 against bn-bpe-64k's 1.241, the one register where the two are not bit-identical. 0.001 fertility on 2,009 sentences is 1-2 tokens' difference in total, not a structural finding; it is reported because it is real and measured, not suppressed for being small.

On the Banglish register (romanized-Bengali chat text, Latin script), this model ranks last of 19 tokenizers measured, by design: BMBT's whole architecture is built around Bengali-script akshara structure, which buys nothing on Latin-script input. This is the real gap that motivated Project Bornomala's separate transliteration pipeline: transliterate Latin-script Banglish to Bengali script first (53.9% exact-match, 13.5% character error rate, trained seq2seq with beam search), then hand off to this tokenizer. Full account: GitHub repository docs/known-issues.md, Banglish section.

Hard words: conjuncts and Bengali place names

A fixed list of 13 deity names, the national poet Rabindranath Tagore, and well-known West Bengal place names, all conjunct-dense, measured on every tokenizer this project tracks (19 total).

This model tokenizes every one of the 13 words as exactly one token, including the triple-conjunct আকাঙ্ক্ষা and the multi-akshara রবীন্দ্রনাথ. BanglaBERT and BanglaT5 (csebuetnlp) also score a perfect 1.00 average here. What sets this model apart: it guarantees the result by construction (the akshara grammar cannot produce a boundary that cuts a conjunct), not by whatever a statistically learned vocabulary happened to cover on these 13 specific words. Full per-word table: benchmarks/hard-words.md.

Usage

BMBT uses an akshara-atom scheme, so encode, decode, and the featural decomposition go through the bntok package's BMBT class.

pip install "bntok @ git+https://github.com/konkomaji/bornomala#subdirectory=bengali-tokenizer"
# or clone the repo and: pip install -e bengali-tokenizer

from huggingface_hub import snapshot_download
from bntok.bmbt import BMBT

path = snapshot_download("konko/bornomala-bengali-tokenizer")
# the config file is stored as bornomala_config.json; rename to config.json,
# or keep all three files (tokenizer.json, atoms.json, config.json) in one directory.
tok = BMBT.load(path)

text = "আমি বাংলায় ক্ষুদ্র গান গাই"
ids = tok.encode(text)
assert tok.decode(ids) == text   # exact round-trip
print(len(ids), "tokens")

for feat in tok.featurize(text):
    print(feat)   # onset consonants, nukta, vowel, trailing modifiers, ZWJ/ZWNJ

To simply count tokens with the raw atom-space model (advanced), load tokenizer.json with the tokenizers library directly, but note it expects atom-remapped input; the bntok.bmbt.BMBT wrapper is the supported path.

Training details

• Algorithm: BPE over akshara atoms, vocabulary 64,000.
• Atomic unit: the akshara found by bntok/akshara.py's finite-state virama parser (a whole conjunct chain), not a UAX #29 grapheme cluster.
• Corpus: literary-weighted, 1.5M lines: Wikisource Bengali public-domain text, AI4Bharat Sangraha verified/ben (pdf-typed as a formal/literary proxy, OCR-noise-filtered, and web-typed for general register), the first 15,000 articles of wikimedia/wikipedia config 20231101.bn, and XL-Sum Bengali news. Identical corpus to the sibling bn-bpe-64k model, so the comparison above isolates the atomization method. Full mix and what was substituted: repository docs/known-issues.md point 6.
• Normalisation: NFC plus explicit re-composition of the three Bengali letters NFC leaves decomposed (ড়/ঢ়/য়), and a preserve-by-default ZWJ/ZWNJ policy.
• Hardware: CPU only.

Limitations

Two of the corpus's six configured sources (government/administrative text, code-mixed Bengali-English) have no clean public dataset and are omitted. The literary/formal register is a proxy (Sangraha pdf-typed documents: genuinely old-orthography and OCR-noisy, but not confirmed pre-1950 public domain). Ranks last of 19 tokenizers measured on Latin-script Banglish input, by design (see Performance above); use the project's transliteration pipeline first for that register. The morphology-aware variant (bmbt-64k-morph) is not the artifact shipped in this repository and is not yet published as a separate Hugging Face model. See the repository's docs/known-issues.md for the full, honest list.

Citation

@software{maji_bornomala_bmbt_2026,
  author  = {Maji, Konko},
  title   = {BMBT: A Grammar-Parsed, Featurally Decomposed Bengali Tokenizer},
  year    = {2026},
  note    = {Project Bornomala. v1 (bmbt-64k)},
  url     = {https://github.com/konkomaji/bornomala}
}

License

Apache 2.0. A Project Bornomala release. Founder: Konko Maji ([email protected]).