vemo-speech-emotion-recognition

VEMO Speech Emotion Recognition

This repository contains the trained speech emotion recognition model developed for VEMO, an AI-powered voice diary with emotion recognition.

The model is designed for Persian speech emotion recognition and predicts four emotion categories:

• Anger
• Happiness
• Neutral
• Sadness

The final system combines deep speech representations extracted from Wav2Vec2 with a set of acoustic/prosodic features and uses a Logistic Regression classifier for the final emotion prediction.

---

Model Overview

The final speech emotion recognition pipeline consists of two phases.

Input Audio
     │
     ▼
Wav2Vec2 Feature Extractor
     │
     ▼
Fine-tuned Wav2Vec2 Encoder
     │
     ▼
Temporal Mean Pooling
     │
     ├─────────────── Deep Speech Representation
     │
     └─────────────── Acoustic / Prosodic Features
                           │
                           ▼
                    Feature Concatenation
                           │
                           ▼
                         Scaler
                           │
                           ▼
                 Logistic Regression
                           │
                           ▼
              Emotion Probabilities

Phase 1 — Speech Encoder

The first phase contains the fine-tuned Wav2Vec2-based speech encoder.

The encoder is based on a Persian speech emotion recognition model built using Wav2Vec2-XLSR and subsequently adapted during the VEMO development process.

The encoder is used to extract a fixed-length deep representation from an input speech signal.

The representation is obtained by applying temporal mean pooling over the Wav2Vec2 hidden states.

Phase 2 — Final Classifier

The second phase contains the components required for the final classification stage:

• Feature scaler
• Logistic Regression classifier
• Metadata

The final classifier receives a concatenation of:

1. Deep speech representation extracted by the Phase 1 Wav2Vec2 encoder
2. Acoustic/prosodic features extracted directly from the audio

The concatenated feature vector is scaled using the stored scaler and passed to the Logistic Regression classifier.

Both Phase 1 and Phase 2 are required for the complete inference pipeline.

---

Repository Structure

vemo-speech-emotion-recognition/
│
├── phase1/
│   ├── config.json
│   ├── model.safetensors
│   ├── preprocessor_config.json
│   └── training_args.bin
│
├── phase2/
│   ├── logistic classifier.joblib
│   ├── scaler.joblib
│   └── metadata.json
│
└── README.md

Phase 1 files

Phase 2 files

---

Emotion Classes

The final system predicts four emotion classes:

---

Feature Extraction

Deep Speech Features

For each input audio file, the Wav2Vec2 encoder produces a sequence of hidden representations.

Temporal mean pooling is applied to obtain a fixed-length representation:

Wav2Vec2 hidden states
        ↓
Temporal mean pooling
        ↓
Fixed-length speech embedding

The Wav2Vec2 encoder is used as a frozen feature extractor during the final classification stage.

Acoustic Features

In addition to the deep Wav2Vec2 representation, eight acoustic features are extracted directly from the input speech:

• Mean fundamental frequency (F0)
• Standard deviation of F0
• F0 range
• Mean RMS energy
• Standard deviation of RMS energy
• Mean zero-crossing rate
• Mean spectral centroid
• Standard deviation of spectral centroid

These features are concatenated with the pooled Wav2Vec2 representation before being passed to the feature scaler and Logistic Regression classifier.

The final feature vector is therefore:

Wav2Vec2 representation
        +
8 acoustic features
        ↓
Concatenated feature vector
        ↓
Feature scaling
        ↓
Logistic Regression

---

Development

The model was developed through multiple stages, including evaluation of a pretrained Wav2Vec2 model, audio augmentation, and adaptation using real-world speech data.

The final system uses a frozen Wav2Vec2 encoder combined with acoustic features and a Logistic Regression classifier.

For the complete development process, experiments, and comparison of approaches, see the GitHub repository.

Python Dependencies

The inference pipeline uses:

torch
transformers
librosa
numpy
joblib

Example

import joblib
import librosa
import numpy as np
import torch

from transformers import Wav2Vec2FeatureExtractor, Wav2Vec2Model


PHASE1_PATH = "path/to/phase1"
PHASE2_PATH = "path/to/phase2"


feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(
    PHASE1_PATH
)

encoder = Wav2Vec2Model.from_pretrained(
    PHASE1_PATH
)

encoder.eval()

scaler = joblib.load(
    f"{PHASE2_PATH}/scaler.joblib"
)

classifier = joblib.load(
    f"{PHASE2_PATH}/logistic_classifier.joblib"
)


def extract_acoustic_features(speech, sr):

    f0, _, _ = librosa.pyin(
        speech,
        fmin=librosa.note_to_hz("C2"),
        fmax=librosa.note_to_hz("C7"),
        sr=sr
    )

    f0_voiced = f0[~np.isnan(f0)]

    if len(f0_voiced) == 0:
        f0_mean = 0.0
        f0_std = 0.0
        f0_range = 0.0
    else:
        f0_mean = float(np.mean(f0_voiced))
        f0_std = float(np.std(f0_voiced))
        f0_range = float(
            np.max(f0_voiced) - np.min(f0_voiced)
        )

    rms = librosa.feature.rms(y=speech)[0]

    rms_mean = float(np.mean(rms))
    rms_std = float(np.std(rms))

    zcr_mean = float(
        np.mean(
            librosa.feature.zero_crossing_rate(speech)[0]
        )
    )

    centroid = librosa.feature.spectral_centroid(
        y=speech,
        sr=sr
    )[0]

    centroid_mean = float(np.mean(centroid))
    centroid_std = float(np.std(centroid))

    return np.array([
        f0_mean,
        f0_std,
        f0_range,
        rms_mean,
        rms_std,
        zcr_mean,
        centroid_mean,
        centroid_std
    ])


def predict_emotion(audio_path):

    speech, _ = librosa.load(
        audio_path,
        sr=feature_extractor.sampling_rate,
        mono=True
    )

    with torch.no_grad():

        inputs = feature_extractor(
            speech,
            sampling_rate=feature_extractor.sampling_rate,
            return_tensors="pt",
            padding=True
        )

        hidden_states = encoder(
            inputs["input_values"]
        ).last_hidden_state

        embedding = (
            hidden_states
            .mean(dim=1)
            .squeeze(0)
            .numpy()
        )

    acoustic_features = extract_acoustic_features(
        speech,
        feature_extractor.sampling_rate
    )

    features = np.concatenate(
        [embedding, acoustic_features]
    ).reshape(1, -1)

    features_scaled = scaler.transform(features)

    probabilities = classifier.predict_proba(
        features_scaled
    )[0]

    return {
        emotion: float(probability)
        for emotion, probability
        in zip(classifier.classes_, probabilities)
    }

The paths in the example should be replaced with the local paths to the downloaded phase1 and phase2 directories.

---

Using the Model with VEMO

This model is the speech emotion recognition component of VEMO.

The VEMO backend loads:

• The Phase 1 Wav2Vec2 encoder
• The Phase 2 scaler
• The Phase 2 Logistic Regression classifier

The backend then processes an input audio file and returns the probability distribution across the four supported emotion classes.

---

Limitations

The model was developed and evaluated using a relatively limited amount of real-world speech data.

Important limitations include:

• Limited number of speakers
• Limited size of the independent test set
• Potential variation in microphones and recording environments
• Potential differences between training and real-world speech conditions
• Performance may vary across speakers, recording devices, and acoustic environments

---

Related Work and References

The initial pretrained speech emotion recognition model used in the development process was:

m3hrdadfi/wav2vec2-xlsr-persian-speech-emotion-recognition

The original Persian speech emotion recognition work is based on the ShEMO dataset:

Mohamad Nezami, O., Jamshid Lou, P., & Karami, M. (2019). ShEMO: A large-scale validated database for Persian speech emotion detection. Language Resources and Evaluation, 53(1), 1–16.

---

VEMO

VEMO is an AI-powered voice diary with emotion recognition and empathetic response generation.

This repository contains the speech emotion recognition model component developed for VEMO.

For the complete VEMO project, including its overall architecture and other components, see the VEMO GitHub repository.