urdu-ser-model

Bilingual Speech Emotion Recognition Model (Urdu + English)

Model Overview

This model performs bilingual Speech Emotion Recognition (SER) from audio input in both Urdu and English languages. It is a fine-tuned version of the multilingual facebook/wav2vec2-xls-r-300m model, trained on a combined dataset of English (RAVDESS + CREMA-D) and Urdu (UrduSER) emotional speech to predict 7 Ekman emotions.

The model is part of the "Multimodal AI Mental Health Companion" Final Year Project (FYP) and is specifically designed for code-switched and multilingual emotional speech analysis in Pakistani contexts where speakers often mix Urdu and English.

Supported Emotions (7 Ekman Classes)

Language Capabilities

---

Model Details

• Model ID: muhammadsuleman1533/urdu-ser-model
• Task: Audio Classification / Speech Emotion Recognition
• Languages: Urdu & English (Bilingual)
• Base Model: facebook/wav2vec2-xls-r-300m (Pre-trained on 128 languages)
• Framework: PyTorch + Hugging Face Transformers
• Model Size: 0.3B Parameters
• Developed by: Muhammad Suleman (Team Leader: Muhammad)
• License: MIT

Why wav2vec2 XLS-R for Bilingual SER?

The XLS-R (300m) model was selected because it is pre-trained on 128 languages, including both Urdu and English. This makes it uniquely suited for:

1. Cross-lingual transfer: Knowledge learned from high-resource English emotional speech (RAVDESS, CREMA-D) transfers to improve Urdu emotion recognition
2. Bilingual robustness: The shared multilingual representations help handle code-switched Urdu-English speech common in urban Pakistani populations
3. Low-resource adaptation: Leverages pre-trained Urdu speech features despite limited Urdu SER data availability

---

Dataset Information

The model was trained on a bilingual hybrid corpus combining English and Urdu emotional speech to maximize generalization for both languages.

Dataset Composition

Language Distribution

Data Split (Speaker-Disjoint)

A GroupShuffleSplit was used based on speaker_id to ensure zero speaker overlap between train and test sets. This tests the model's true ability to recognize emotion in unseen voices across both languages.

Emotion Distribution & Class Imbalance

The combined dataset is moderately imbalanced with surprise being heavily under-represented in both languages.

Important Notes:

• Surprise class: Only present in English datasets (RAVDESS). UrduSER does not contain surprise samples.
• Mitigation: Weighted Cross-Entropy Loss was used with surprise weighted 8.5x higher to compensate for extreme under-representation.

---

Training Details

Training Configuration

Freezing Strategy

To prevent catastrophic forgetting of pre-trained multilingual speech representations:

• Frozen: CNN feature extractor layers (wav2vec2.feature_extractor)
• Trainable: Transformer encoder layers + Classification head

Preprocessing Pipeline

1. Audio Loading: All files loaded with librosa at 16kHz mono
2. Duration Filtering: Files outside 0.5-10 seconds filtered out (removed 6 corrupted files)
3. Feature Extraction: Wav2Vec2FeatureExtractor from facebook/wav2vec2-xls-r-300m
4. Label Encoding: 7 emotions mapped to numeric IDs (0-6)
5. Standardization: All emotions mapped to 7 Ekman classes (e.g., "calm" → "neutral", "happy" → "joy")

---

Evaluation Metrics

Overall Performance (Final Model)

The model was trained for 10 epochs with early stopping (patience=3). Training converged at epoch 6.7.

Per-Language Performance (Estimated)

Since the test set contains both Urdu and English samples without separate labels, these are conservative estimates based on dataset composition:

Class-wise Performance (Actual Results)

The following table shows the actual per-class performance on the 2,757 test samples:

| | Weighted Avg | 0.583 | 0.573 | 0.554 | 2,757 |

Key Observations

Strong Performance (F1 > 0.60):

• Anger (0.774): Excellent recall (88%) - the model rarely misses anger when present. High intensity and distinct prosodic features make this emotion easily recognizable across both languages.
• Neutral (0.635): Very high recall (84%) - the model effectively identifies non-emotional speech, though precision is moderate due to some confusion with joy.
• Surprise (0.634): Despite having only 32 test samples (all English), the model correctly identifies all surprise samples (100% recall), though precision is lower as it sometimes misclassifies fear as surprise.

Moderate Performance (F1 0.45-0.60):

• Joy (0.556): Moderate performance with balanced precision and recall. Some confusion with neutral speech.
• Fear (0.520): Decent precision (66%) but lower recall (43%) - the model misses many fear samples, likely confusing them with surprise or sadness.

Weak Performance (F1 < 0.45):

• Sadness (0.479): Good precision (69%) but very low recall (37%) - the model is conservative in predicting sadness, missing many true sadness samples.
• Disgust (0.325): The most challenging emotion. Low recall (28%) indicates the model struggles to distinguish disgust from anger, which shares similar acoustic properties.

Training Progress

The model showed consistent improvement throughout training, with the best F1-weighted score (0.554) achieved at step 4500 (epoch ~5.6).

---

How to Use the Model

Installation

pip install transformers torch librosa soundfile