nllb-medical-ablation-masked

NLLB-200 + LoRA — Medical English → Urdu (Entity-Masked Ablation Variant)

Model Details

Model Description

This model is the ablation variant of ayeshasadiq025/nllb-medical-clinical. Both models share the exact same architecture and hyperparameters — the only difference is the training input:

The ablation study was designed to answer: does replacing medical entity names with placeholder tokens before translation help or hurt the model?

Result: the baseline (raw text, no masking) outperformed this masked variant — confirming that entity masking hurt translation quality by preventing the model from learning to produce actual Urdu medical terminology.

• Developed by: Ayesha Sadiq (BSE-25S-007), SMIU, Karachi
• Supervised by: Sir Amin Chhajro, Department of Software Engineering, SMIU
• Model type: Sequence-to-Sequence (Encoder-Decoder), Transformer
• Languages: English (eng_Latn) → Urdu (urd_Arab)
• License: Apache 2.0
• Fine-tuned from: facebook/nllb-200-distilled-600M
• Fine-tuning method: LoRA (r=8, alpha=32) via 🤗 PEFT
• Purpose: Ablation study — not recommended for production use. Use nllb-medical-clinical instead.

Model Sources

• Baseline model (better, recommended): ayeshasadiq025/nllb-medical-clinical
• Other models in this project:
  • mT5 fine-tuned: ayeshasadiq025/mt5-medical-urdu
  • BioBERT + Decoder: ayeshasadiq025/biobert-medical-urdu-decoder

---

⚠️ Important Note

This model is an ablation/research variant — not the production model.

It was trained on masked input text (medical entities replaced with placeholder tokens). At inference time, you must also mask the input in the same way before passing it to this model — otherwise outputs will be unpredictable.

For standard use, please use ayeshasadiq025/nllb-medical-clinical which was trained on raw text and gives better results.

---

Uses

Direct Use

Research and comparison purposes only. This model demonstrates the effect of entity masking on translation quality for low-resource medical NLP.

Out-of-Scope Use

• Production medical translation (use the baseline model instead)
• Input without entity masking applied (this model expects masked input)
• Urdu → English direction
• Clinical decision-making without expert review

---

Bias, Risks, and Limitations

• Masked input required: This model was fine-tuned on masked_english_text. If you pass raw text without applying the entity masking preprocessing step, outputs will be lower quality.
• Worse than baseline: The ablation study confirmed this model performs worse than the raw-text baseline on all automatic and human evaluation metrics.
• Not clinically validated: Should not be used in safety-critical medical workflows without expert review.
• Machine-generated Urdu references: Training Urdu translations were initially generated by a translation utility and post-processed.

---

How to Get Started with the Model

⚠️ You must apply entity masking to your input text before passing it to this model.

import re
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_name = "ayeshasadiq025/nllb-medical-ablation-masked"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.float16)
model.eval()

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

# Required: entity masking preprocessing (same as used during training)
import spacy
nlp = spacy.load("en_core_web_sm")

def mask_medical_entities(text: str) -> str:
    """Replace medical entities with placeholder tokens — must match training preprocessing."""
    doc = nlp(text)
    masked = text
    for ent in reversed(doc.ents):
        if ent.label_ in ["DISEASE", "CHEMICAL", "DRUG", "ORG"]:
            masked = masked[:ent.start_char] + "[MEDICAL_TERM]" + masked[ent.end_char:]
    return masked

def translate(text: str) -> str:
    # Step 1: apply entity masking
    masked_text = mask_medical_entities(text)

    # Step 2: tokenize and translate
    inputs = tokenizer(
        masked_text,
        return_tensors="pt",
        max_length=128,
        truncation=True
    ).to(device)

    target_lang_id = tokenizer.convert_tokens_to_ids("urd_Arab")

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            forced_bos_token_id=target_lang_id,
            max_new_tokens=128,
            num_beams=4,
            early_stopping=True
        )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)


text = "Patient was diagnosed with acute myocardial infarction."
print(translate(text))

---

Training Details

Training Data

Same 12,500-sentence parallel corpus as the baseline model, but using the masked_english_text column (medical entities replaced with placeholder tokens during spaCy NER preprocessing):

Preprocessing — Entity Masking Pipeline

The key difference from the baseline model:

1. Abbreviation expansion: ICU, BP, MRI, HIV, etc. expanded to full forms
2. spaCy NER: en_core_web_sm used to detect medical entities (diseases, chemicals, drugs, procedures)
3. Entity replacement: Detected entities replaced with typed placeholders:
   • Disease entities → [DISEASE]
   • Chemical/drug entities → [DRUG_CHEMICAL]
   • Other medical entities → [MEDICAL_TERM]
4. This masked text (not the raw expanded text) was used as the training source

Training Hyperparameters

Identical to the baseline model nllb-medical-clinical:

---

Evaluation — Ablation Study Results

The ablation study evaluated both variants on the same 100-sentence held-out test sample (random_state=42). Results confirmed that entity masking hurt performance — the baseline without masking was consistently better:

Why did masking hurt?

The original hypothesis was that masking medical entity names before translation would help preserve them in the Urdu output. In practice, the opposite occurred — when the model was trained on text with placeholder tokens ([MEDICAL_TERM], [DISEASE]), it learned to produce those placeholders in the Urdu output instead of actual Urdu medical terms. This reduced both fluency and faithfulness of translations.

The baseline model, trained on raw text with full medical entity names, learned to directly map English medical terms to their Urdu equivalents — producing more natural and medically accurate output.

Summary

The ablation study established that for NLLB-200+LoRA fine-tuning on this dataset, raw (non-masked) input text produces better medical translation quality than entity-masked input. This is an important empirical finding for medical NLP practitioners considering entity masking as a preprocessing strategy for low-resource translation.

---

Environmental Impact

• Hardware: NVIDIA T4 GPU (Google Colab free tier)
• Cloud Provider: Google Colab
• Training duration: 3 epochs (~a few hours on T4)
• Precision: FP16 mixed precision
• Carbon estimate: ML CO2 Impact Calculator

---

Technical Specifications

Model Architecture

Identical to the baseline:

• Base: facebook/nllb-200-distilled-600M — 600M parameter multilingual seq2seq Transformer
• Adaptation: LoRA adapters on q_proj and v_proj attention layers (encoder + decoder)
• Inference: LoRA weights merged into base model before upload — standard AutoModelForSeq2SeqLM API

Software

transformers
peft
torch
spacy (en_core_web_sm — for input masking at inference)
sentencepiece
protobuf
sacrebleu
bert-score
rouge_score
datasets

---

Citation

@misc{sadiq2026nllbablation,
  title        = {Ablation Study: Effect of Medical Entity Masking on NLLB-200+LoRA English-to-Urdu Translation},
  author       = {Ayesha Sadiq},
  year         = {2026},
  institution  = {Sindh Madressatul Islam University (SMIU), Karachi},
  note         = {BSE-25S-007. Supervised by Amin Chhajro. Ablation variant of https://huggingface.co/ayeshasadiq025/nllb-medical-clinical}
}

---

Model Card Authors

Ayesha Sadiq — Department of Software Engineering, SMIU, Karachi Supervisor: Sir Amin Chhajro

Model Card Contact

HuggingFace: ayeshasadiq025