whisper-large-v3-SER

🎙️ Whisper-SER (IEMOCAP + MEACorpus)

Multilingual Speech Emotion Recognition for Iberian Languages A Generative AI Approach with LLMs and Data Augmentation Techniques

---

🧠 Overview

This repository hosts the model whisper-ser-iemocap_meacorpus.ckpt, released as part of the paper:

Bellver-Soler, J., Guragain, A., Ramos-Varela, S., Córdoba, R., & D’Haro, L.F. (2025) Multilingual Speech Emotion Recognition for Iberian Languages: A Generative AI Approach with LLMs and Data Augmentation Techniques. Universidad Politécnica de Madrid (UPM) 📄 SSRN Preprint

This model combines Whisper-large-v3 as a self-supervised acoustic encoder with attentive pooling and a frozen Bloomz-7B1 LLM classifier, trained for emotion recognition on the IEMOCAP and MEA-Corpus datasets.

---

🚀 Model Description

---

🎯 Task

Speech Emotion Recognition (SER) Predict the emotion expressed in an audio waveform among:

['neutral', 'happy', 'sad', 'angry', 'fear', 'surprise', 'disgust']

---

🧩 Training Details

• Stratified speaker-independent splits (train/val/test)
• 5 random seeds for robustness
• Validation metric: F1-macro (95% CI)
• Training in bfloat16 on 2× NVIDIA A100 (40GB)

---

📈 Performance

Frozen Bloomz-7B1 improves F1 by +4.9% over a linear MLP head.

---

🧬 Architecture

Audio → Whisper Encoder → Attentive Pooling → Linear Projection → Bloomz-7B1 (frozen) → Emotion logits

---

🧪 Usage Example

import torch
import torchaudio

# Load model checkpoint
model = torch.load("whisper-ser-iemocap_meacorpus.ckpt", map_location="cpu")
model.eval()

# Example audio
waveform, sr = torchaudio.load("example.wav")
assert sr == 16000, "Audio must be 16 kHz"

# Forward pass
with torch.no_grad():
    logits = model(waveform)
    probs = torch.nn.functional.softmax(logits, dim=-1)
    emotion = probs.argmax(dim=-1)

print("Predicted emotion:", emotion.item())

---

🧾 Citation

If you use this model, please cite:

@article{bellversoler2025multilingual,
  title={Multilingual Speech Emotion Recognition for Iberian Languages: A Generative AI Approach with LLMs and Data Augmentation Techniques},
  author={Bellver-Soler, Jaime and Guragain, Anmol and Ramos-Varela, Samuel and Córdoba, Ricardo and D’Haro, Luis Fernando},
  journal={Computer Speech & Language},
  year={2025},
  note={Preprint, SSRN: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5244228}
}

---

📂 Related Resources

---

📜 License

Released under CC BY 4.0. Attribution required for derivative works. Note: MEACorpus includes YouTube-sourced content — additional rights may apply.

---

🙌 Acknowledgements

Supported by:

• European Commission – ASTOUND3 (101071191, Horizon Europe)
• MCIN/AEI/ERDF – Project BEWORD (PID2021-126061OB-C43)
• INNOVATRAD-CM – Comunidad de Madrid (PHS-2024/PH-HUM-52)

---

Authors: Jaime Bellver-Soler, Anmol Guragain, Samuel Ramos-Varela, Ricardo Córdoba, Luis Fernando D’Haro Speech Technology and Machine Learning Group, Universidad Politécnica de Madrid