finetuned-en-bn-nllb

Model Card for nllb-finetuned-en-bn

This is a fine-tuned version of Meta's NLLB-200 (600M) model, optimized specifically for English-Bengali conversational translation. Unlike standard translation models that focus on formal text, this model is trained on daily conversation datasets to provide more natural chat translations.

Model Details

Model Description

• Developed by: Sifat
• Model type: Neural Machine Translation (NLLB/M2M100 architecture)
• Language(s) (NLP): English (eng_Latn) <-> Bengali (ben_Beng)
• License: MIT
• Finetuned from model: facebook/nllb-200-distilled-600M

Model Sources

• Repository: https://huggingface.co/SifatAlMukit/finetuned-en-bn-nllb
• Dataset: Tatoeba Project (English-Bengali pairs)

Uses

Direct Use

This model is designed for:

• Chat Applications: Translating messages between English and Bengali users.
• Mobile Apps: Offline or API-based translation features.
• Learning: Helping users understand daily spoken sentences.

Out-of-Scope Use

• Legal/Medical Documents: The model is not trained on technical terminology.
• Long Articles: Performance may degrade on very long paragraphs as it is optimized for sentence-level translation.

Bias, Risks, and Limitations

• Vocabulary: Since it is fine-tuned on ~7,000 conversational pairs, it might struggle with rare or highly complex words.
• Context: Like most NMT models, it may occasionally misinterpret the gender or context if it's ambiguous in the source sentence.

How to Get Started with the Model

You can use this model with the Hugging Face transformers library.

Example 1:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load model and tokenizer
model_name = "SifatAlMukit/finetuned-en-bn-nllb"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

def translate(text, direction="en2bn"):
    # Set configuration based on direction
    if direction == "en2bn":
        tokenizer.src_lang = "eng_Latn"
        target_lang = "ben_Beng"
    else:
        tokenizer.src_lang = "ben_Beng"
        target_lang = "eng_Latn"

    # Tokenize and Generate
    inputs = tokenizer(text, return_tensors="pt")
    generated_tokens = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(target_lang),
        max_length=100
    )
    
    # Decode result
    result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
    return result

# Example Usage
print(translate("This AI Model can be used as a translator", "en2bn"))
# Output: আপনি কেমন আছেন?

print(translate("এই মডেলটিকে বাংলায় ফাইন টিউন করা হয়েছে।", "bn2en"))
# Output: Please close the door.

print(translate("এই মডেলটিকে ২০২৬ সালের একদিন আগে প্রকাশ করা হয়েছে। অর্থাৎ, এটিকে ৩১ ডিসেম্বর ২০২৫ তারিখে প্রকাশ করা হয়েছে।", "bn2en"))

Example 2:

# ১. ভাষার কোডগুলো মনে রাখা কঠিন, তাই একটা ডিকশনারি বানিয়ে নিলাম
lang_codes = {
    "English": "eng_Latn",
    "Bengali": "ben_Beng",
    "Hindi":   "hin_Deva",
    "Spanish": "spa_Latn",
    "French":  "fra_Latn",
    "German":  "deu_Latn",
    "Chinese": "zho_Hans", # Simplified Chinese
    "Arabic":  "arb_Arab",
    "Japanese": "jpn_Jpan",
    "Russian": "rus_Cyrl"
}

# ২. মাল্টি-ল্যাঙ্গুয়েজ টেস্ট ফাংশন
def test_multilingual(text, source_lang_name, target_lang_names):
    src_code = lang_codes.get(source_lang_name)
    
    print(f"\n🔹 Input ({source_lang_name}): {text}")
    print("-" * 60)
    
    for target_name in target_lang_names:
        tgt_code = lang_codes.get(target_name)
        
        if src_code and tgt_code:
            # আপনার মূল translate ফাংশনটি কল করা হচ্ছে
            translated_text = translate(text, src_lang=src_code, tgt_lang=tgt_code)
            print(f"{target_name:<15} | {translated_text}")
        else:
            print(f"Error: Language code not found for {target_name}")
    print("=" * 60)

# ৩. টেস্ট কেস (Test Cases)
# ফরম্যাট: (বাক্য, সোর্স ভাষা, [কোন কোন ভাষা থেকে ট্রান্সলেট করা হবে তার লিস্ট])

# কেস ১: ইংরেজি থেকে ইউরোপিয়ান ভাষা
test_multilingual(
    "Technology is changing the world very fast.", 
    "English", 
    ["Bengali", "Spanish", "French", "German"]
)

# কেস ২: ইংরেজি থেকে এশিয়ান ভাষা (জটিল স্ক্রিপ্ট টেস্ট করার জন্য)
test_multilingual(
    "I love to learn new programming languages.", 
    "English", 
    ["Hindi", "Chinese", "Japanese", "Arabic"]
)

# কেস ৩: বাংলা থেকে অন্য ভাষা
test_multilingual(
    "আজকের আকাশটা খুব নীল।", 
    "Bengali", 
    ["English", "Hindi", "Spanish"]
)

# কেস ৪: প্রশ্নবোধক বাক্য
test_multilingual(
    "Where is the nearest subway station?", 
    "English", 
    ["Bengali", "French", "Japanese"]
)

Example 3:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

# Load model and tokenizer
model_name = "SifatAlMukit/finetuned-en-bn-nllb"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Use GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

def translate(text, src_lang="eng_Latn", tgt_lang="ben_Beng"):
    """
    Translate text using optimized generation parameters.
    """
    tokenizer.src_lang = src_lang
    
    # Tokenize and move to device
    inputs = tokenizer(text, return_tensors="pt").to(device)
    
    # Generate translation with parameters to fix repetition issues
    generated_tokens = model.generate(
        **inputs,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(tgt_lang),
        max_length=100,
        num_beams=5,             # Uses beam search for better quality
        no_repeat_ngram_size=2,  # Prevents repeating 2-word phrases
        repetition_penalty=1.5,  # Discourages repetitive text
        early_stopping=True
    )
    
    # Decode result
    result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
    return result

# Example Usage

# 1. English to Bengali
english_text = "Where is the nearest subway station?"
print(f"English: {english_text}")
print(f"Bengali: {translate(english_text, src_lang='eng_Latn', tgt_lang='ben_Beng')}")
# Output: কাছাকাছি সাবওয়ে স্টেশন কোথায়?

# 2. Bengali to English
bengali_text = "দরজাটি বন্ধ করুন।"
print(f"Bengali: {bengali_text}")
print(f"English: {translate(bengali_text, src_lang='ben_Beng', tgt_lang='eng_Latn')}")
# Output: Please close the door.

Training Details

Training Data

The model was fine-tuned on the Tatoeba English-Bengali dataset, containing approximately 7,000 high-quality sentence pairs focusing on daily life interactions.

Training Procedure

• Framework: PyTorch & Transformers
• Hardware: Trained on T4 GPU (Google Colab)
• Epochs: 3
• Batch Size: 8
• Learning Rate: 2e-5

Environmental Impact

• Hardware Type: T4 GPU
• Hours used: ~1 hour
• Cloud Provider: Google Colab
• Carbon Emitted: Negligible