CF_training_modal

A Unified Approach for Multimodal Emotion Recognition Using Counterfactual Learning

📋 Overview

This repository contains the official implementation of "A Unified Approach for Multimodal Emotion Recognition Using Counterfactual Learning", a novel framework designed to enhance multimodal emotion recognition through advanced sequential modeling and structured counterfactual learning.

Key Features

• 🎯 State-of-the-Art Performance: Achieves 94.87% weighted F1-score on MELD and 86.68% on IEMOCAP datasets
• 🔄 Three-Phase Training: Progressive counterfactual learning with baseline training, alignment, and intention-guided refinement
• 🧠 Advanced Architecture: Progressive BiLSTM modules with cross-attention fusion for comprehensive multimodal integration
• 📊 Robust Evaluation: Comprehensive analysis with ROC curves, confusion matrices, and t-SNE visualizations
• ⚡ Efficient Processing: Optimized for GPU training with mixed-precision support

Architecture Highlights

• Modality-Specific Feature Extraction: RoBERTa (text), Wav2Vec2 (audio), ResNet-50 (visual)
• Progressive BiLSTM: Hierarchical temporal feature modeling with adaptive gating
• Cross-Modal Attention: All-pairs bidirectional attention for rich inter-modality fusion
• Counterfactual Learning: Novel three-phase training strategy for enhanced robustness

📊 Data

Datasets

The framework is evaluated on two benchmark datasets for conversational emotion recognition:

MELD (Multimodal Emotion Lines Dataset)

• Source: Multi-party dialogues from Friends TV series
• Emotions: 7 classes (anger, disgust, fear, joy, neutral, sadness, surprise)
• Modalities: Text, audio, visual
• Download: MELD Dataset

IEMOCAP (Interactive Emotional Dyadic Motion Capture)

• Source: Dyadic interactions between actors
• Emotions: 8 classes (angry, excited, fear, frustrated, happy, neutral, sad, surprised)
• Modalities: Text, audio, visual
• Download: IEMOCAP Dataset

Data Structure

data/
├── MELD/
│   ├── train/
│   ├── dev/
│   └── test/
└── IEMOCAP/
    ├── train/
    ├── dev/
    └── test/

🚀 Installation/Setup

Prerequisites

• Python 3.8+
• CUDA-capable GPU (recommended)
• 16GB+ RAM

Clone Repository

git clone https://github.com/BLANK-2340/A-Unified-Approach-for-Multimodal-Emotion-Recognition-Using-Counterfactual-Learning.git
cd A-Unified-Approach-for-Multimodal-Emotion-Recognition-Using-Counterfactual-Learning

Install Dependencies

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install required packages
pip install -r requirements.txt

# Install additional dependencies for audio processing
pip install librosa soundfile

# Install pre-trained models
python -c "from transformers import RobertaTokenizer, Wav2Vec2Processor; RobertaTokenizer.from_pretrained('roberta-base'); Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base-960h')"

Environment Setup

# Set environment variables
export CUDA_VISIBLE_DEVICES=0
export TOKENIZERS_PARALLELISM=false

🎯 Usage

Quick Start

# Train the model with default settings
python train.py --dataset MELD --batch_size 16 --epochs 40

# Evaluate trained model
python evaluate.py --model_path checkpoints/best_model.pth --dataset MELD

# Generate predictions
python predict.py --input_file data/sample.csv --output_file results/predictions.csv

Training Configuration

# Custom training with specific parameters
python train.py \
    --dataset MELD \
    --batch_size 16 \
    --learning_rate 5e-5 \
    --weight_decay 0.01 \
    --phase1_epochs 15 \
    --phase2_epochs 15 \
    --phase3_epochs 10 \
    --save_dir checkpoints/

Inference

from model import MultimodalEmotionRecognizer
from utils import load_data, preprocess

# Load pre-trained model
model = MultimodalEmotionRecognizer.from_pretrained('checkpoints/best_model.pth')

# Process input data
text, audio, video = preprocess(input_data)

# Predict emotion
prediction = model.predict(text, audio, video)
print(f"Predicted emotion: {prediction}")

Pre-trained Model

You can download our pre-trained model checkpoint from the following Google Drive link:

Download Pre-trained Model

This checkpoint can be used for inference as demonstrated in the Inference section.

🔬 Methodology

Our approach integrates advanced sequential modeling with a novel three-phase counterfactual learning strategy to address key challenges in multimodal emotion recognition.

1. Overall Architecture

<div align="center"> <img src="Architecture%20of%20Multimodal%20Emotion%20Recognition%20Model.png" alt="Architecture" width="800"/> </div>

Architecture Overview: The framework processes synchronized audio, visual, and textual data through three main components:

1. Modality-Specific Feature Extraction: Each modality (text, audio, video) is processed through dedicated pipelines:

   • Text: RoBERTa tokenization → contextual embeddings → Feature Transformation Block (FTB)
   • Audio: Wav2Vec2 voice embeddings + MFCC + spectral features → self-attention → combination
   • Video: Entropy-based frame selection → ResNet-50 → BiLSTM → multi-head attention

2. Progressive BiLSTM Processing: Each modality's features undergo hierarchical temporal modeling with adaptive gating mechanisms that progressively refine representations across multiple layers.

3. Cross-Modal Fusion: All-pairs bidirectional cross-attention mechanism facilitates rich information exchange between modalities, followed by standardization and MLP-based classification.

4. Counterfactual Feature Generation (CFG): Generates minimally perturbed feature versions to enhance model robustness through the three-phase training strategy.

2. Cross-Attention Mechanism

<div align="center"> <img src="Cross%20Attention%20Block%20for%20Feature%20Enhancement;%20CMA.png" alt="Cross Attention" width="600"/> </div>

Cross-Modal Attention (CMA) enables bidirectional information exchange across all modality pairs. The mechanism computes attention weights between query features from one modality and key-value pairs from another:

Q_{m_i} = W_Q^{m_1} F_{m_i}, \quad K_{m_j} = W_K^{m_2} F_{m_j}, \quad V_{m_j} = W_V^{m_2} F_{m_j}

A_{m_i \rightarrow m_j} = \text{softmax}\left(\frac{Q_{m_i} K_{m_j}^T}{\sqrt{256}}\right) V_{m_j}

Process Flow:

1. Six Attention Pairs: (audio→video), (audio→text), (video→audio), (video→text), (text→audio), (text→video)
2. Enhanced Features: Each modality's original features are concatenated with cross-attended features from other modalities
3. Standardization: Features undergo layer normalization and dimensionality reduction for stable fusion

3. Three-Phase Counterfactual Training Strategy

<div align="center"> <img src="Illustration%20of%20Three-Phase%20Counterfactual%20Training.png" alt="Three-Phase Training" width="700"/> </div>

The training strategy progressively incorporates counterfactual reasoning to enhance model robustness:

Phase 1: Base Model Training

• Establishes baseline representations using standard multimodal fusion
• Loss function focuses on accurate emotion classification using Focal Loss to handle class imbalance

\mathcal{L}_{Phase1} = \mathcal{L}_{Focal}(\hat{y}_{P1}, y_{True})

Phase 2: Counterfactual Alignment

• Introduces counterfactual feature generation and alignment loss
• Encourages consistency between original and counterfactual representations of the same emotion class
• Uses contrastive learning to push apart different emotion classes

\mathcal{L}_{Phase2} = \mathcal{L}_{Focal}(\hat{y}_{P2}, y_{True}) + \lambda_{align}(e) \cdot \mathcal{L}_{align}(H_{org}, H_{cf}, y_{True})

Phase 3: Intention-Guided Refinement

• Activates the Intention Predictor module for explicit counterfactual relationship modeling
• Focuses on high-confidence samples to refine decision boundaries
• Combines all three loss components with adaptive scheduling

\mathcal{L}_{Phase3} = \mathcal{L}_{Focal} + \lambda_{align} \cdot \mathcal{L}_{align} + \lambda_{intent}(e) \cdot \mathcal{L}_{intent}

4. Intention Predictor Module

<div align="center"> <img src="Illustration%20of%20Intention%20Predictor%20module.png" alt="Intention Predictor" width="500"/> </div>

Purpose: Models the relationship between original and counterfactual features to understand prediction rationale.

Architecture:

1. Difference Computation: Calculates feature difference between counterfactual and original representations
2. Concatenation: Combines difference vector with original features for context
3. MLP Processing: Three-layer MLP with dropout and layer normalization for intention prediction

H_{diff} = H_{cf} - H_{org}

\tilde{H}_{diff} = [H_{diff}; H_{org}] \in \mathbb{R}^{B \times 512}

\hat{y}^{int} = \text{MLP}(\tilde{H}_{diff})

Intention Loss: Confidence-weighted cross-entropy with label smoothing, focusing on samples where the base model is already confident (confidence > 0.7).

5. Mathematical Formulations

Progressive BiLSTM with Adaptive Gating:

H_{lstm}^{(i)} = \text{BiLSTM}^{(i)}(F^{(i-1)}) \in \mathbb{R}^{B \times SeqLen \times 2h}

H_{attn}^{(i)} = \text{MultiHeadAttn}(H_{lstm}^{(i)}, H_{lstm}^{(i)}, H_{lstm}^{(i)}) \in \mathbb{R}^{B \times SeqLen \times 2h}

G^{(i)} = \sigma(\text{Linear}(\text{Concat}(T^{(i)}, F^{(i-1)})))

F^{(i)} = G^{(i)} \odot T^{(i)} + (1 - G^{(i)}) \odot F^{(i-1)}

Alignment Loss (InfoNCE-based):

S_{ij} = \frac{\langle \hat{F}_i, \hat{F}_{cf,j} \rangle}{\tau}

\mathcal{L}_{align} = -\text{mean}_i \left( \log \frac{\exp(\sum_{j \in Pos(i)} S_{ij})}{\exp(\sum_{j \in Pos(i)} S_{ij}) + \exp(\sum_{k \in Neg(i)} [\max(S_{ik} + m, 0)])} \right)

Where τ = 0.1 (temperature), m = 0.5 (margin), and Pos(i)/Neg(i) represent positive/negative sample sets.

📈 Results

Performance Comparison

Our framework achieves state-of-the-art results on both benchmark datasets:

Training Progress Analysis

<div align="center"> </div>

Learning Curve Analysis:

• Phase 1 (Epochs 1-15): Rapid initial learning as the model establishes baseline multimodal representations
• Phase 2 (Epochs 16-30): Steady improvement with counterfactual alignment, showing enhanced feature discriminability
• Phase 3 (Epochs 31-40): Fine-tuning through intention prediction, achieving optimal performance on MELD

ROC Curves Across Training Phases

MELD Dataset

<table align="center"> <tr> <td align="center"><b>Phase 1</b></td> <td align="center"><b>Phase 2</b></td> <td align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="roc_curves_phase1_MELD.png" width="250"/></td> <td><img src="roc_curves_phase2_MELD.png" width="250"/></td> <td><img src="roc_curves_phase3_MELD.png" width="250"/></td> </tr> </table>

IEMOCAP Dataset

<table align="center"> <tr> <td align="center"><b>Phase 1</b></td> <td align="center"><b>Phase 2</b></td> <td align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="roc_curves_phase1_iemocap.png" width="250"/></td> <td><img src="roc_curves_phase2_iemocap.png" width="250"/></td> <td><img src="roc_curves_phase3_iemocap.png" width="250"/></td> </tr> </table>

ROC Analysis:

• Progressive improvement in discriminative power across training phases
• Phase 3 MELD: AUC values exceeding 0.95 for all emotion classes, with near-perfect scores (AUC ≈ 1.0) for disgust and fear
• Phase 2 IEMOCAP: Optimal balance achieved with strong AUCs (>0.96) for most emotions
• Class-specific insights: Fear and surprise consistently show excellent discrimination, while neutral emotions remain challenging across datasets

Confusion Matrix Evolution

MELD Dataset

<table align="center"> <tr> <td align="center"><b>Phase 1</b></td> <td align="center"><b>Phase 2</b></td> <td align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="custom_confusion_matrix_phase1_MELD.png" width="250"/></td> <td><img src="custom_confusion_matrix_phase2_MELD.png" width="250"/></td> <td><img src="custom_confusion_matrix_phase3_MELD.png" width="250"/></td> </tr> </table>

IEMOCAP Dataset

<table align="center"> <tr> <td align="center"><b>Phase 1</b></td> <td align="center"><b>Phase 2</b></td> <td align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="custom_confusion_matrix_phase1_iemocap.png" width="250"/></td> <td><img src="custom_confusion_matrix_phase2_iemocap.png" width="250"/></td> <td><img src="custom_confusion_matrix_phase3_iemocap.png" width="250"/></td> </tr> </table>

Confusion Matrix Insights:

• Diagonal Strengthening: Clear improvement in diagonal concentration from Phase 1 to Phase 3
• Error Reduction: Significant reduction in off-diagonal misclassifications, particularly between similar emotions
• Challenging Pairs: Neutral-joy and frustration-anger confusions persist but decrease substantially
• Perfect Classification: Some emotions (fear, disgust) achieve near-perfect classification in final phases

Feature Space Visualization with t-SNE

Cross-Attention Output Features (MELD)

<table align="center"> <tr> <td align="center"><b>Phase 1</b></td> <td align="center"><b>Phase 2</b></td> <td align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="tsne_cross_attention_phase1_MELD.png" width="250"/></td> <td><img src="tsne_cross_attention_phase2_MELD.png" width="250"/></td> <td><img src="tsne_cross_attention_phase3_MELD.png" width="250"/></td> </tr> </table>

Cross-Attention t-SNE Analysis:

• Phase 1: Significant overlap between emotion clusters, indicating insufficient discriminative power
• Phase 2: Introduction of alignment loss creates more compact and distinct clusters for each emotion
• Phase 3: Clearest separation achieved with intention-guided refinement, showing well-defined emotion boundaries

Progressive BiLSTM Output Features (MELD)

<table align="center"> <tr> <td colspan="3" align="center"><b>Phase 1</b></td> </tr> <tr> <td align="center">Layer 1</td> <td align="center">Layer 2</td> <td align="center">Layer 3</td> </tr> <tr> <td><img src="tsne_bilstm_layer1_phase1_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer2_phase1_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer3_phase1_MELD.png" width="250"/></td> </tr> <tr> <td colspan="3" align="center"><b>Phase 2</b></td> </tr> <tr> <td><img src="tsne_bilstm_layer1_phase2_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer2_phase2_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer3_phase2_MELD.png" width="250"/></td> </tr> <tr> <td colspan="3" align="center"><b>Phase 3</b></td> </tr> <tr> <td><img src="tsne_bilstm_layer1_phase3_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer2_phase3_MELD.png" width="250"/></td> <td><img src="tsne_bilstm_layer3_phase3_MELD.png" width="250"/></td> </tr> </table>

Progressive BiLSTM t-SNE Analysis:

• Hierarchical Learning: Each BiLSTM layer progressively refines feature representations
• Layer Evolution: From Layer 1 to Layer 3, emotion clusters become increasingly well-separated
• Phase Progression: Counterfactual training (Phases 2-3) significantly enhances cluster formation across all layers
• Temporal Dynamics: The sequential nature of BiLSTM processing creates increasingly abstract and discriminative features

Per-Class Performance Analysis

MELD Dataset (Phase 3)

IEMOCAP Dataset (Phase 2)

Performance Insights:

• Excellent Recognition: Fear, disgust, and surprise achieve near-perfect performance due to distinct multimodal signatures
• Challenging Classes: Neutral and frustration show lower performance due to subtle expression patterns and overlap with other emotions
• Multimodal Advantage: High-arousal emotions (anger, surprise) benefit significantly from combined audio-visual cues
• Dataset Differences: MELD shows more balanced performance across classes, while IEMOCAP exhibits greater variation

📋 File Structure

├── Counterfactual_Training_Run/     # Main training scripts and implementations
│   ├── train.py                     # Training pipeline
│   ├── model.py                     # Model architecture
│   └── losses.py                    # Loss functions
├── Video_vector/                    # Video feature extraction modules
│   ├── frame_extraction.py         # Entropy-based frame selection
│   ├── resnet_features.py          # ResNet-50 feature extraction
│   └── video_lstm.py               # BiLSTM for video sequences
├── Audio_vector/                    # Audio feature processing modules
│   ├── wav2vec_features.py         # Wav2Vec2 voice embeddings
│   ├── mfcc_extraction.py          # MFCC coefficient extraction
│   └── spectral_features.py        # Spectral feature computation
├── Text_vector/                     # Text feature processing modules
│   ├── roberta_features.py         # RoBERTa text embeddings
│   └── text_preprocessing.py       # Text preprocessing utilities
├── images/                          # Visualization results and figures
│   ├── Architecture diagrams/      # Model architecture illustrations
│   ├── ROC curves/                 # ROC analysis across phases
│   ├── Confusion matrices/         # Classification performance matrices
│   ├── t-SNE visualizations/       # Feature space visualizations
│   └── Learning curves/            # Training progress plots
├── model/                           # Core model architecture components
│   ├── progressive_bilstm.py       # Progressive BiLSTM implementation
│   ├── cross_attention.py          # Cross-modal attention mechanism
│   ├── counterfactual_generator.py # CFG module
│   └── intention_predictor.py      # Intention prediction module
├── utils/                           # Utility functions and helpers
│   ├── data_loader.py              # Dataset loading and preprocessing
│   ├── metrics.py                  # Evaluation metrics
│   └── visualization.py            # Plotting and visualization tools
├── config/                          # Configuration files
│   ├── meld_config.yaml            # MELD dataset configuration
│   └── iemocap_config.yaml         # IEMOCAP dataset configuration
├── requirements.txt                 # Python dependencies
└── README.md                        # Project documentation

💡 Key Contributions

1. Novel Progressive Architecture: Modality-specific Progressive BiLSTM with internal self-attention and adaptive gating mechanisms for hierarchical temporal modeling that captures complex intra-modal dynamics

2. Three-Phase Counterfactual Learning: Structured training strategy incorporating:

   • Phase 1: Baseline multimodal fusion establishment
   • Phase 2: Counterfactual alignment for enhanced feature discriminability
   • Phase 3: Intention-guided refinement for explicit counterfactual relationship modeling

3. Comprehensive Cross-Modal Fusion: All-pairs bidirectional attention mechanism enabling rich information exchange between text, audio, and visual modalities through six distinct attention pathways

4. State-of-the-Art Performance: Achieving significant improvements over existing methods:

   • 21.55% improvement in F1-score on MELD dataset
   • 12.77% improvement in F1-score on IEMOCAP dataset

5. Robust Feature Learning: Enhanced discriminability and generalization through counterfactual reasoning, demonstrated via comprehensive t-SNE visualizations showing progressive cluster formation

🔧 Technical Specifications

• Framework: PyTorch 1.9.0+ with CUDA support
• Model Size: 105.3M trainable parameters
• Computational Complexity: 23.31 GFLOPs
• Optimization: AdamW optimizer with phase-specific learning rate scheduling
• Training Features:
  • Mixed-precision training (FP16) for memory efficiency
  • Gradient clipping (max norm: 1.0) for stability
  • Early stopping with patience monitoring
  • Automatic gradient accumulation

📞 Contact/Authors

Armaan Singh
Department of Electronics and Communication Engineering
Delhi Technological University, Delhi, India

Chhavi Dhiman
Department of Electronics and Communication Engineering
Delhi Technological University, Delhi, India

---

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Delhi Technological University for computational resources and research support
• The creators of MELD and IEMOCAP datasets for providing benchmark evaluation platforms
• The open-source community for valuable tools, libraries, and pre-trained models
• The research community for foundational work in multimodal emotion recognition and counterfactual learning

---

<div align="center">

⭐ Star this repository if you find it helpful!

</div>