nllb-eng-idoma

NLLB-200 fine-tuned for English <-> Idoma

Idoma (ISO 639-3 idu, Glottolog idom1241) is a Benue-Congo language of Benue State, Nigeria. Stock NLLB-200 does not support it — its tokenizer has 202 language codes and idu_Latn is not among them, so idu_Latn resolves to <unk> and the model returns the source sentence unchanged.

This checkpoint adds idu_Latn as a real token, resizes the embedding matrix, and initialises the new row from ibo_Latn (Igbo, the nearest in-vocabulary relative) before fine-tuning.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("emoduh/nllb-eng-idoma")
model = AutoModelForSeq2SeqLM.from_pretrained("emoduh/nllb-eng-idoma")

# This must NOT be the unk id.
tgt = tok.convert_tokens_to_ids("idu_Latn")
assert tgt != tok.unk_token_id

tok.src_lang = "eng_Latn"
batch = tok("Come and eat", return_tensors="pt")
out = model.generate(**batch, forced_bos_token_id=tgt, max_length=96, num_beams=4)
print(tok.decode(out[0], skip_special_tokens=True))

Training data

Roughly 989 English-Idoma pairs, predominantly word-level, derived from the community dictionary at https://www.idomaland.org/dictionary with gratitude to its contributors. The dataset itself is not redistributed here.

Entries are dialect-tagged where the source distinguishes them (central vs western Idoma differ: water is Ennkpo in central and Enyi in western).

Limitations

• Trained on a small, mostly word-level corpus: treat it as a dictionary-augmented translator, not a fluent sentence translator.
• Tone marking is inconsistent in the source data. Tone is contrastive in Idoma (àkpà "bridge" vs ákpá "cloud"), so some outputs will be tonally wrong.
• Dialect coverage is uneven; central Idoma dominates.
• Do not use for anything safety-critical.

Evaluation

chrF++ on a held-out split, keyed by English headword so dialect variants of the same word cannot leak between train and test. See the training notebook (training/train_idoma_nllb.ipynb) for exact numbers from your run.