wavlm-large-emotion-vad

wavlm-large-emotion-vad — dimensional speech emotion (Valence / Arousal / Dominance)

WavLM-large fine-tuned for dimensional emotion regression on naturalistic speech: audio in → estimated Valence / Arousal / Dominance ratings out. Training updated the transformer backbone and regression head while freezing the convolutional feature extractor. The model predicts annotation-defined perceived affect, not a verified internal emotional state. The two-phase project record, including earlier handcrafted-feature baselines, is at github.com/Nikhiljangra07/voice-emotion-engine.

Headline result

Reported evaluation on MSP-Podcast v2.0 Test1, using the dataset's official speaker-independent split (approximately 46k clips). Metric: Concordance Correlation Coefficient (CCC). These values are recorded in this repository's train.log:

The project investigated valence after earlier handcrafted-feature stages reported CCC 0.06 and 0.35. This checkpoint reported 0.705 on Test1. Those stages used different experimental setups, so the sequence is a development history rather than a controlled effect-size estimate. It does not establish a general ceiling for acoustic features or a state-of-the-art result.

What's in this repo

Usage

This repository contains the backbone and a separate regression-head state dict. Loading the backbone alone does not produce V/A/D predictions. Download config.json, model.safetensors, and head.pt into the same local checkpoint directory.

The published inference implementation shows the exact head construction, mean pooling, and preprocessing: mono 16 kHz audio, peak normalization, and truncation to the first eight seconds. The three training targets are MSP-Podcast SAM ratings mapped from 1–7 to 0–1. Its display conversion clips outputs to 0–1 before mapping them back to SAM/PAD coordinates.

The script's full CLI also loads separately generated emotion-naming artifacts from models/namer_msp_final; those artifacts are not included in this Hugging Face repository. Consult the project documentation for those dependencies. The regression checkpoint itself contains no categorical emotion-naming layer. No new inference run was performed for this documentation update.

Training data & license

Trained on MSP-Podcast v2.0 (Lotfian & Busso; Busso et al. 2025, arXiv:2509.09791) under its academic license. The dataset is not included and cannot be redistributed — request access from UT-Dallas. This repository retains its existing CC-BY-NC-SA 4.0 (non-commercial) license. That label does not replace the dataset's separate access and use terms or confer rights to its recordings.

Honest scope

• Evaluation is on English, podcast-style speech. Earlier project experiments found transfer limitations between acted and naturalistic speech; this is not a guarantee of performance across languages, recording conditions, or populations.
• Dominance is the weakest measured axis in this run (0.626).
• Single checkpoint and training run, with no ensembling or reported confidence intervals.
• Test1 counts are approximate: the training/evaluation loader skips unreadable or very short files, and the published log does not report the exact scored Test1 count.
• The training/evaluation implementation averages hidden states over padded sequences. Reproducing the numbers requires matching that implementation and batching; changing pooling or preprocessing needs a fresh evaluation.

---

Documentation updated 2026-10-27 to clarify methods and limitations. Reported results and checkpoint files are unchanged; no new training or evaluation was performed for this update.