bn-mni-transliteration-ct2

Bengali script → Meetei Mayek transliteration (Manipuri)

A character-level Transformer that transliterates Manipuri (Meiteilon) written in Bengali script into the Meetei Mayek script. Trained with OpenNMT-py and shipped here in CTranslate2 format for fast CPU inference.

👉 Try it in the browser: Space

Usage

import ctranslate2
from huggingface_hub import snapshot_download

path = snapshot_download("Prabuddha14/bn-mni-transliteration-ct2")
translator = ctranslate2.Translator(path, device="cpu")

word = "এনর্জি"
result = translator.translate_batch([list(word)], beam_size=5)
print("".join(result[0].hypotheses[0]))   # ꯑꯦꯅꯔꯖꯤ

The model takes one word at a time, as a list of its characters. To run over running text, use the transliterate.py helper from the Space, which handles sentence splitting, batching, caching and the details below.

Input handling that matters

Getting these wrong noticeably degrades output:

1. Normalise, then recompose nukta letters. ড় ঢ় য় are Unicode composition exclusions, so unicodedata.normalize("NFC", …) splits them into base + U+09BC. The model's vocabulary holds the precomposed forms, so they must be joined back up after normalising.
2. Do not send digits to the model. Digits are rare in the training data and decode badly (২০২৪ → ꯲꯲꯲꯲꯲꯲꯲꯲). Map them arithmetically instead: Bengali ০–৯ (U+09E6–U+09EF) → Meetei Mayek ꯰–꯹ (U+ABF0–U+ABF9).
3. Pass unknown characters through. Punctuation, Latin text and symbols are outside the source vocabulary; copying them verbatim is better than letting them become <unk>. The danda । maps to the cheikhei ꯫.

Evaluation

On a 100-sentence internal test set (test.bn / ref.mni), beam size 5:

CER is computed with jiwer over the concatenated lines. The gain comes from preserving punctuation, mapping digits directly, and recomposing nukta letters — the model weights are identical in both rows.

Limitations

• Digits and Latin text are not learned; they are handled by rules, not the model.
• Trained on a specific domain (government / news-style prose). Conversational or dialectal text is out of distribution.
• Transliteration is ambiguous in places: Bengali script under-specifies some Meetei Mayek distinctions, so a small residual error rate is inherent.
• No sentence-level context — each word is decoded independently.

Training data

<!-- TODO: describe the parallel corpus (source, size, licence) before making this repo public. -->

Citation

<!-- TODO: add the paper or project citation. -->

Developed at CFILT, IIT Bombay.