bohok-indic-en

Bohok Indic → English (vivekharry/bohok-indic-en)

Marian MT (Helsinki-NLP/opus-mt-mul-en) fine-tuned one epoch on CPU so you can load a real checkpoint with transformers and translate Kokborok, Bengali, and Marathi into English.

This is not a universal translator and not Google Translate. It is a small many-to-English model adapted on:

Held-out split: 4% (~1360 sentences). Train set: 32,657. One CPU epoch (4,083 steps, 42m). Train loss 2.49, eval loss 2.05.

Held-out examples after training (not cherry-picked UI strings):

Kokborok is the low-resource track. Bengali/Marathi inherit OPUS-100. This is not Google Translate.

Prefixes (required)

The tokenizer gained three special tokens. Always prefix the source:

>>trp<< <kokborok text>
>>bn<<  <bengali text>
>>mr<<  <marathi text>

Kokborok training data is mostly Latin / Roman SMOL spelling, plus some Bangla-script items from GATITOS. Roman Kokborok works better than free dialect you invent.

Use it

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "vivekharry/bohok-indic-en"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSeq2SeqLM.from_pretrained(repo)

def to_en(text, lang="trp"):
    ids = tok(f">>{lang}<< {text}", return_tensors="pt", truncation=True, max_length=128)
    out = model.generate(**ids, max_new_tokens=96, num_beams=4)
    return tok.decode(out[0], skip_special_tokens=True)

print(to_en("আমার দিকে তাকাও.", "bn"))
print(to_en("टॉम जेवतोय.", "mr"))

CLI from this repo:

python scripts/translate.py --model vivekharry/bohok-indic-en --lang bn "আজ আকাশটা খুব সুন্দর।"

Limits (read this before a viva)

• One epoch on a 77M Marian, CPU, ~30k mixed sentences. Bengali/Marathi inherit OPUS quality; they will still fail on long literary text.
• Kokborok coverage is SMOL-scale. Unseen morphology, tone, and dialect will degrade.
• Direction is into English only.
• Speech is a separate stack (Whisper / MMS). This repo is the MT checkpoint.

Train the same recipe

python scripts/prepare_bohok_data.py
python scripts/train_bohok_mt.py --epochs 1 --bs 8 --out checkpoints/bohok-indic-en
python scripts/eval_and_push.py --repo vivekharry/bohok-indic-en

Author: Vivek Das (vivekharry).