wav2vec2-base_speech_emotion_recognition

Fine-Tuned Wav2Vec2 for Speech Emotion Recognition

Model Details

Model Name: Fine-Tuned Wav2Vec2 for Speech Emotion Recognition
Base Model: facebook/wav2vec2-base
Dataset: narad/ravdess
Quantization: Available as an optional FP16 version for optimized inference
Training Device: CUDA (GPU)

Dataset Information

Dataset Structure:

DatasetDict({
    train: Dataset({
        features: ['audio', 'text', 'labels', 'speaker_id', 'speaker_gender'],
        num_rows: 1440
    })
})

Note: Split manually into 80% train (1,152 examples) and 20% validation (288 examples) during training, as the original dataset provides only a single "train" split.

Available Splits:

• Train: 1,152 examples (after 80/20 split)
• Validation: 288 examples (after 80/20 split)
• Test: Not provided; external audio used for testing

Feature Representation:

• audio: Raw waveform (48kHz, resampled to 16kHz during preprocessing)
• text: Spoken sentence (e.g., "Dogs are sitting by the door")
• labels: Integer labels for emotions (0–7)
• speaker_id: Actor identifier (e.g., "9")
• speaker_gender: Gender of speaker (e.g., "male")

Training Details

• Number of Classes: 8

• Class Names: neutral, calm, happy, sad, angry, fearful, disgust, surprised

• Training Process: Fine-tuned for 10 epochs (initially 3, revised to 10 for better convergence)

• Learning rate: 3e-5, with warmup steps (100) and weight decay (0.1) -Batch size: 4 with gradient accumulation (effective batch size 8)

• Dropout added (attention_dropout=0.1, hidden_dropout=0.1) for regularization

• Performance Metrics

• Epochs: 10

• Training Loss: ~0.8

• Validation Loss: ~1.2

• Accuracy: ~0.65

• F1 Score: ~0.63

Inference Example

import torch
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2Processor
import librosa

def load_model(model_path):
    model = Wav2Vec2ForSequenceClassification.from_pretrained(model_path)
    processor = Wav2Vec2Processor.from_pretrained(model_path)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    model.eval()
    return model, processor, device

def predict_emotion(model_path, audio_path):
    model, processor, device = load_model(model_path)
    
    # Load and preprocess audio
    audio, sr = librosa.load(audio_path, sr=16000)
    inputs = processor(audio, sampling_rate=16000, return_tensors="pt", padding=True, max_length=160000, truncation=True)
    input_values = inputs["input_values"].to(device)
    
    # Inference
    with torch.no_grad():
        outputs = model(input_values)
        logits = outputs.logits
        predicted_label = torch.argmax(logits, dim=1).item()
        probabilities = torch.softmax(logits, dim=1).squeeze().cpu().numpy()
    
    emotions = ['neutral', 'calm', 'happy', 'sad', 'angry', 'fearful', 'disgust', 'surprised']
    return emotions[predicted_label], {emotion: prob for emotion, prob in zip(emotions, probabilities)}

# Example usage
if __name__ == "__main__":
    model_path = "path/to/wav2vec2-ravdess-emotion/final_model"  # Update with your HF username/repo
    audio_path = "path/to/audio.wav"
    emotion, probs = predict_emotion(model_path, audio_path)
    print(f"Predicted Emotion: {emotion}")
    print("Probabilities:", probs)

Quantization & Optimization

• Quantization: Optional FP16 version created using PyTorch’s .half() for faster inference with reduced memory footprint.
• Optimized: Suitable for deployment on GPU-enabled devices; FP16 version reduces model size by ~50%.

Usage

• Input: Raw audio files (.wav) resampled to 16kHz
• Output: Predicted emotion label (one of 8 classes) with confidence probabilities

Limitations

• Generalization: Trained on acted speech (RAVDESS), may underperform on spontaneous or noisy real-world audio.
• Dataset Size: Limited to 1,440 samples, potentially insufficient for robust emotion recognition across diverse conditions.
• Accuracy: Performance on external audio varies; retraining with augmentation or larger datasets may be needed.

Future Improvements

• Data Augmentation: Incorporate noise, pitch shift, or speed changes to improve robustness.
• Larger Dataset: Combine with additional SER datasets (e.g., IEMOCAP, CREMA-D) for diversity.
• Model Tuning: Experiment with freezing lower layers or using a model pre-trained for SER (e.g., facebook/wav2vec2-large-robust).