xlm-roberta-multilingual-emotion-classifier

xlm-roberta-multilingual-emotion-classifier

Model Description

This model is a fine‑tuned version of XLM‑RoBERTa‑base for 7‑class emotion detection in code‑mixed Pakistani text, including English, Roman Urdu, and Urdu script. It is designed for social media analysis, customer feedback, and multilingual NLP applications in the Pakistani context.

• Base Model: xlm-roberta-base
• Emotion Classes: anger, disgust, fear, joy, neutral, sad, surprise
• Language Support: English, Roman Urdu, Urdu script

Training Data & Balancing Strategy

Data Sources

The training dataset was compiled from multiple sources:

Cleaning & Preprocessing

• Duplicate rows removed
• Standardised column structure across three languages (English, Urdu, Roman Urdu)
• Seven emotion classes retained
• Light spelling normalisation for Roman Urdu

Class Imbalance & Balancing Strategy

Initial raw data showed significant imbalance:

• English: Fear and Sad dominated (~14-16), Disgust very low (~7.1%).
• Roman Urdu: Fear as low as 8.1%, Sad relatively low.
• Urdu: More balanced but Sad often lowest.

Targeted Augmentation (Weak Classes)

We augmented Sad, Disgust, Surprise (and Fear/Joy where needed per language) using:

• Random Swap (EDA) for Urdu and Roman Urdu – swaps two random words, preserving sentiment while increasing structural variety.
• Oversampling + Cross‑Lingual Injection for English Disgust – duplicated high‑quality samples and added code‑switched (English + Roman Urdu) variants.
• LLM‑based translation (GPT, Grok, Gemini) to generate parallel English‑Urdu/Roman Urdu versions.

Approximate additional samples added:

• Total synthetic/augmented samples: ~2,000–2,500
• English Disgust: +334 samples (~30% of that class augmented)
• Urdu/Roman Urdu weak classes (Surprise, Disgust): +400–500 samples each

Majority Class Control (Mild Downsampling)

We randomly under‑sampled the majority English classes (Anger, Fear, Sad) to prevent bias:

• Original counts: ~1,400–1,600 each
• Reduced by ~200–400 samples to a ceiling of ~1,200 each

Quality Controls

• Augmentation applied to the master dataset before train/test split (so test set also contains augmented variants, proving model robustness).
• Stratified split ensures balanced representation across languages and emotions.
• Manual spot‑check performed on augmented Roman Urdu samples.

Final Dataset Statistics

Total rows: 20,235
Languages: English (7,734), Roman Urdu (5,572), Urdu (6,929)

Per‑language emotion counts (exact numbers used in training):

Global percentages:

• Highest: Fear (~15.34%)
• Lowest: Surprise (~12.63%)
• Others: 13.8% – 14.9%

The final dataset is much more balanced than raw sources, with all classes represented across all three languages.

Training Details

• Framework: PyTorch + Hugging Face Transformers
• Model: XLM‑RoBERTa‑base (12 layers, 278M parameters)
• Optimizer: AdamW (learning rate = 2e‑5)
• Batch size: 32
• Epochs: 5 (early stopping patience = 2)
• Max sequence length: 128 tokens
• Warmup steps: 10% of total steps
• Weight decay: 0.01
• FP16: Enabled (GPU acceleration)

Evaluation Results

The model was evaluated on a held‑out test set of 4,047 samples (20% of total data, stratified).

Overall Performance

• Accuracy: 83.4%
• Macro F1‑score: 0.835
• Weighted F1‑score: 0.833

Per‑Class Performance

Confusion Matrix (Actual vs. Predicted)

Per‑Language Accuracy