all-indic-transliteration

All Indic Languages Transliteration

Convert between English and 21 Indian languages instantly. Type namaste → get नमस्ते, or reverse it. No PyTorch. No Fairseq. Just one pip install and you're running.

This is a production-ready, optimized version of AI4Bharat's IndicXlit — converted to CTranslate2 for maximum speed and minimum hassle.

---

Why use this instead of the original?

The official IndicXlit repo requires installing PyTorch, Fairseq, and multiple dependencies that frequently break on modern Python versions. We solved all of that.

---

What's Included

Two ready-to-use CTranslate2 models (FP32 precision):

Each folder contains:

model.bin              ← The neural network weights
config.json            ← CTranslate2 engine configuration  
source_vocabulary.json ← Input character vocabulary
target_vocabulary.json ← Output character vocabulary

---

Installation

pip install ctranslate2 huggingface_hub

That's it. No PyTorch, no Fairseq, no CUDA required.

---

Usage

English → Hindi (Forward)

Type in English letters, get back native Hindi script:

import ctranslate2
from huggingface_hub import snapshot_download
import os

# Download the model (only happens once, cached after that)
model_path = snapshot_download(
    repo_id="Singla0009/all-indic-transliteration",
    allow_patterns="indicxlit_ct2_fp32/*"
)
model_dir = os.path.join(model_path, "indicxlit_ct2_fp32")

# Load the engine
translator = ctranslate2.Translator(model_dir, device="cpu")

# Transliterate (Output: नमस्ते)
word = "namaste"
source_tokens = ["__hi__"] + list(word)
results = translator.translate_batch([source_tokens], beam_size=4)
output = "".join(results[0].hypotheses[0])

print(f"{word} → {output}")

Hindi → English (Reverse)

Take native Devanagari script, get back romanized English:

import ctranslate2
from huggingface_hub import snapshot_download
import os

# Download the reverse model
model_path = snapshot_download(
    repo_id="Singla0009/all-indic-transliteration",
    allow_patterns="indicxlit_indic_en_ct2_fp32/*"
)
model_dir = os.path.join(model_path, "indicxlit_indic_en_ct2_fp32")

# Load the engine
translator = ctranslate2.Translator(model_dir, device="cpu")

# Transliterate (Output: namaste)
word = "नमस्ते"
source_tokens = ["__hi__"] + list(word)
results = translator.translate_batch([source_tokens], beam_size=4)
output = "".join(results[0].hypotheses[0])

print(f"{word} → {output}")

Batch Processing (Multiple Words)

For real-world use, process many words at once for maximum throughput:

words = ["namaste", "dhanyavaad", "bharat", "cricket", "youtube"]
batch = [["__hi__"] + list(w) for w in words]

results = translator.translate_batch(batch, beam_size=4)

for word, result in zip(words, results):
    output = "".join(result.hypotheses[0])
    print(f"{word} → {output}")

Switching Languages

Just change the language prefix. That's it — same model, same code:

# Tamil
source_tokens = ["__ta__"] + list("vanakkam")

# Bengali  
source_tokens = ["__bn__"] + list("namaskar")

# Punjabi
source_tokens = ["__pa__"] + list("sat sri akal")

---

Supported Languages (21)

---

Technical Details

• Original Model: AI4Bharat IndicXlit v1.0 (Transformer, 6 encoder + 6 decoder layers, ~11M parameters)
• Conversion Format: CTranslate2 FP32 (lossless — identical output to the original)
• Architecture: Sequence-to-sequence Transformer with character-level tokenization
• Inference Engine: CTranslate2 v4.x (optimized C++ with CPU vectorization)
• Beam Search: Default beam_size=4 for best accuracy (reduce to 1 for maximum speed)

Bugs We Fixed During Conversion

Converting the original Fairseq checkpoints to CTranslate2 required patching several compatibility issues:

1. Python 3.12 dataclass errors — Fairseq uses mutable default arguments in dataclasses, which Python 3.12+ rejects. We patched configs.py and transformer_config.py.
2. OmegaConf/Hydra initialization crash — We disabled the hydra_init() call in Fairseq's __init__.py that causes validation errors during conversion.
3. PyTorch 2.6+ weight loading — Added weights_only=False to torch.load() calls since modern PyTorch blocks legacy pickle loading by default.
4. Vocabulary alignment — Fixed the lang_list.txt to correctly match all 21 language tokens present in the model weights.

---

Use Cases

• Subtitle Romanization — Convert Hindi/Tamil/Bengali subtitles into readable English characters for Hinglish audiences
• Search & Indexing — Enable multilingual search across Indic content by normalizing to a common script
• Input Method Engines — Power transliteration keyboards and text input tools
• Content Localization — Automatically adapt content between scripts for different regional audiences
• NLP Pipelines — Pre/post-processing step for machine translation, ASR output cleanup, and text normalization

---

Credits & License

This is a converted and optimized redistribution of the AI4Bharat IndicXlit model, originally trained on the Aksharantar dataset containing 26 million transliteration pairs.

Released under the MIT License. Original model by the AI4Bharat team at IIT Madras.

If you find this useful, please consider starring the repository! ⭐