nllb-dhivehi-english-cleaned-best

NLLB Dhivehi-English Translation Model (Cleaned)

This model is a fine-tuned version of facebook/nllb-200-distilled-600M for bidirectional translation between Dhivehi Latin script (div_Latn) and English (eng_Latn).

Model Details

Training

Curriculum Learning Strategy

The model was trained using a length-based curriculum approach, starting with short sequences (16 tokens) and progressively increasing to longer sequences (up to 512 tokens).

Total training time: ~2 hours on NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)

Data Pipeline

1. Combined words, sentences, and paragraphs from multiple sources
2. Cleaned aggressively — removed duplicates, HTML, religious content, Arabic script, constitutional text
3. Filtered to max 512 tokens
4. Split into 6 token-length buckets

Final dataset: ~527K rows (down from ~940K after cleaning)

Key Training Fixes

• Language tokens preserved in labels — critical for NLLB convergence
• DataCollatorForSeq2Seq without model= argument
• Forced BOS token per eval batch for correct multilingual generation
• torch.compile disabled — avoids seq2seq gradient issues
• label_smoothing_factor=0.0 — avoids decoder_input_ids conflict in Transformers 5.9

Usage

Python

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained(
    "nllb-dhivehi-english-cleaned-best",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("nllb-dhivehi-english-cleaned-best")

# Dhivehi → English
text = "shukuriyaa"
tokenizer.src_lang = "div_Latn"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, forced_bos_token_id=256047)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# English → Dhivehi
text = "thank you"
tokenizer.src_lang = "eng_Latn"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, forced_bos_token_id=256204)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

CLI

python -c "
from transformers import pipeline
translator = pipeline('translation', model='nllb-dhivehi-english-cleaned-best', src_lang='div_Latn', tgt_lang='eng_Latn')
print(translator('shukuriyaa'))
"

Language Tokens

Performance

Good Translations (Words/Short Phrases)

Known Limitations

⚠️ Sentence-level translation is weak. The model struggles with:

• Repetition loops on longer sentences (kurumash kurumash...)
• Hallucinations (e.g., "Delhi" instead of "Maldives")
• Incomplete grammatical structure

Root causes identified:

1. Insufficient clean conversational sentence data
2. Bad source/target pairs surviving cleaning in buckets 128+
3. Bucket 512 had too few samples and caused overfitting

Best use case: Word-level and short phrase translation. For sentences, the model needs more high-quality training data.

Files

Citation

@misc{nllb2022,
  title = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  author = {NLLB Team, Meta AI},
  year = {2022}
}

License

MIT

Acknowledgments

• Base model: facebook/nllb-200-distilled-600M
• Framework: Hugging Face Transformers
• Training infrastructure: RunPod RTX PRO 6000 Blackwell