wav2vec2-large-xlsr-53-italian-emotion-recognition

🎵 Wav2Vec2.0 Italian Discrete Emotion Classifier

📖 Description

This model is a discrete emotion classifier that assigns an incoming audio segment to one of seven emotion classes: anger, disgust, fear, joy, neutrality, sadness, and surprise.
These correspond to the six basic emotions defined by Ekman~\cite{ekman1992basic} plus the neutral class.

The model uses mean pooling (pooling_mode = mean) over the contextualized representations from the Wav2Vec2.0 encoder, followed by a linear classification head projecting to a 7-dimensional space.

🎯 Intended Use

• Automatic recognition of discrete emotions from Italian speech.
• Prosodic and emotional analysis for research or industrial applications.
• Generating emotion labels for downstream applications such as dialogue systems or affective computing.

⚠️ Not suitable for:

• Languages other than Italian without retraining.
• Clinical or therapeutic use without proper validation.

🚀 Usage Example

from pathlib import Path
import time
import torch
import json
import torch.nn as nn
import torch.nn.functional as F
import torchaudio
import soundfile as sf 
from transformers import AutoConfig, Wav2Vec2FeatureExtractor 
from transformers.file_utils import ModelOutput #type: ignore
from dataclasses import dataclass
from typing import Optional, Tuple
from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
from transformers.models.wav2vec2.modeling_wav2vec2 import (
    Wav2Vec2PreTrainedModel,
    Wav2Vec2Model
)

@dataclass
class SpeechClassifierOutput(ModelOutput):
    loss: Optional[torch.FloatTensor] = None
    logits: torch.FloatTensor = None
    hidden_states: Optional[Tuple[torch.FloatTensor]] = None
    attentions: Optional[Tuple[torch.FloatTensor]] = None


class Wav2Vec2ClassificationHead(nn.Module):
    """Head for wav2vec classification task."""

    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.dropout = nn.Dropout(config.final_dropout)
        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)

    def forward(self, features, **kwargs):
        x = features
        x = self.dropout(x)
        x = self.dense(x)
        x = torch.tanh(x)
        x = self.dropout(x)
        x = self.out_proj(x)
        return x


class Wav2Vec2ForSpeechClassification(Wav2Vec2PreTrainedModel):
    def __init__(self, config, **kwargs):
        super().__init__(config)

        self.all_tied_weights_keys = {}

        self.num_labels = config.num_labels
        self.pooling_mode = config.pooling_mode
        self.config = config

        self.wav2vec2 = Wav2Vec2Model(config)
        self.classifier = Wav2Vec2ClassificationHead(config)

        self.init_weights()

    def freeze_feature_extractor(self):
        self.wav2vec2.feature_extractor._freeze_parameters()

    def merged_strategy(
            self,
            hidden_states,
            mode="mean"
    ):
        if mode == "mean":
            outputs = torch.mean(hidden_states, dim=1)
        elif mode == "sum":
            outputs = torch.sum(hidden_states, dim=1)
        elif mode == "max":
            outputs = torch.max(hidden_states, dim=1)[0]
        else:
            raise Exception(
                "The pooling method hasn't been defined! Your pooling mode must be one of these ['mean', 'sum', 'max']")

        return outputs

    def forward(
            self,
            input_values,
            attention_mask=None,
            output_attentions=None,
            output_hidden_states=None,
            return_dict=None,
            labels=None,
    ):
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )
        hidden_states = outputs[0]
        hidden_states = self.merged_strategy(hidden_states, mode=self.pooling_mode)
        logits = self.classifier(hidden_states)

        loss = None
        if labels is not None:
            if self.config.problem_type is None:
                if self.num_labels == 1:
                    self.config.problem_type = "regression"
                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
                    self.config.problem_type = "single_label_classification"
                else:
                    self.config.problem_type = "multi_label_classification"

            if self.config.problem_type == "regression":
                loss_fct = MSELoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels)
            elif self.config.problem_type == "single_label_classification":
                loss_fct = CrossEntropyLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            elif self.config.problem_type == "multi_label_classification":
                loss_fct = BCEWithLogitsLoss()
                loss = loss_fct(logits, labels)

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return SpeechClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )


model_path = 'voiceAnalysis/model'
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Loading Prosody model from '{model_path}' su {device}...")

config = AutoConfig.from_pretrained(model_path)
setattr(config, 'pooling_mode', 'mean')

feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(model_path)
model = Wav2Vec2ForSpeechClassification.from_pretrained(model_path, config=config)
model.to(device)
model.eval()
    

def speech_file_to_array_fn(audio_path, target_sampling_rate=16000):
    speech_array, src_sampling_rate = sf.read(audio_path)
    
    # From Numpy to Tensor Float
    speech_array = torch.from_numpy(speech_array).float()
    
    if speech_array.ndim == 1:
        speech_array = speech_array.unsqueeze(0) # (1, Time)
    else:
        speech_array = speech_array.t() # (Channels, Time)
        
    # If stereo, mean to mono
    if speech_array.shape[0] > 1:
        speech_array = torch.mean(speech_array, dim=0, keepdim=True)

    if src_sampling_rate != target_sampling_rate:
        resampler = torchaudio.transforms.Resample(src_sampling_rate, target_sampling_rate)
        speech_array = resampler(speech_array)
    
    speech = speech_array.squeeze().numpy()
    return speech


def get_emotion_info(audio_path):
    speech = speech_file_to_array_fn(audio_path)
    
    inputs = feature_extractor(speech, sampling_rate=16000, return_tensors="pt", padding=True)
    inputs = {key: inputs[key].to(device) for key in inputs}

    with torch.no_grad():
        logits = model(**inputs).logits

    scores = F.softmax(logits, dim=1).detach().cpu().numpy()[0]
    
    outputs = []
    for i, score in enumerate(scores):
        outputs.append({
            "Emotion": model.config.id2label[i],
            "Score": round(float(score), 4) 
        })
        
    outputs.sort(key=lambda x: x["Score"], reverse=True)
    return outputs


def main():
    audio_path = "<your_file_path>"

    if Path(audio_path).exists():
        try:
            start_time = time.time()
            results = get_emotion_info(audio_path)
            end_time = time.time()
            
            json_output = json.dumps(results, indent=4)
            print(json_output)
            print(f'/nExecution Time : {end_time - start_time} s')
            
        except Exception as e:
            print(json.dumps({"status": "error", "message": str(e)}))
    else:
        print(json.dumps({"status": "error", "message": "File non trovato"}))
    
if __name__ == "__main__":
    main()

📊 Training Data

The classifier was trained on a combined Italian corpus from three datasets: Emozionalmente, EMOVO, and AI4SER.
This aggregation improves stylistic and demographic coverage and mitigates the small size of individual datasets.

• Training set: 30,878 samples
• Validation set: 3,860 samples

🛠️ Training Configuration

📈 Metrics and Performance

At the best checkpoint (epoch 10, step 2000) the model achieved:

• Evaluation accuracy: 66.81% on 3,860 validation samples
• Evaluation loss: 0.988
• Training loss: 1.009 on 30,878 samples (~19.41 samples/s, total ~4h25m)

💡 Interpretation:

• The small gap between training and evaluation loss suggests no significant overfitting.
• The moderate accuracy reflects the intrinsic difficulty of the task on multi-corpus heterogeneous Italian datasets, where stylistic and demographic differences introduce labeling noise.

⚠️ Limitations

• Trained only on Italian datasets: does not generalize to other languages.
• Performance may vary with audio quality and speaker style.
• Labels are discrete predictions; emotional intensity and nuance are not captured.

📝 License

Apache 2.0

📚 Citation

If you use this model, cite the original repository:

@misc{bernardini2024wav2vec2_emotion,
  author = {Alessio Bernardini},
  title = {Wav2Vec2.0 Italian Discrete Emotion Classifier},
  year = {2026},
  howpublished = {\url{https://github.com/abernardini-unimi/Wav2Vec2.0-Italian-emotion}}
}

🔗 Code and Model Links

• GitHub repository: Wav2Vec2.0-Italian-emotion
• Hugging Face: EmotionLayer – Discrete Emotion Prediction