multimodal-sentiment-model-with-augmentation

Multimodal Sentiment Model with Augmentation

DeBERTa-v3-Large 기반 멀티모달 감성 분석 모델 (CMU-MOSEI)

Model Description

이 모델은 CMU-MOSEI 데이터셋에서 멀티모달 감성 분석을 위해 학습되었습니다. IITP "나비효과" 연구 프로젝트의 일환으로 개발되었습니다.

Architecture

• Text Encoder: DeBERTa-v3-Large (microsoft/deberta-v3-large)
• Audio Encoder: Transformer Encoder (2 layers)
• Video Encoder: Transformer Encoder (2 layers)
• Fusion: Cross-modal attention + Multi-head self-attention

Key Features

• Cross-modal attention between text, audio, and video
• Mixup augmentation for audio/video modalities
• Multi-task learning with auxiliary classifiers (T, A, V branches)
• Frozen first 20 layers of DeBERTa for efficient training

Performance

Comparison with Baselines

Training Details

• Dataset: CMU-MOSEI (unaligned_50.pkl)
• Batch Size: 16
• Learning Rate: 2e-5 (other), 5e-6 (DeBERTa)
• Epochs: 50 (early stopping: 15)
• Optimizer: AdamW
• Scheduler: Cosine with warmup
• Mixup: alpha=0.4, prob=0.5
• Loss weights: cls=0.7, aux=0.1

Usage

import torch
from transformers import AutoTokenizer

# Load checkpoint
checkpoint = torch.load('best_model.pt')
args = checkpoint['args']

# Initialize model
from train_deberta_multimodal import DeBERTaMultimodalModel

model = DeBERTaMultimodalModel(
    model_name='microsoft/deberta-v3-large',
    hidden_size=512,
    num_heads=8,
    dropout=0.2,
    freeze_deberta_layers=20
)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v3-large')

Input Format

• Text: Raw text string (tokenized by DeBERTa tokenizer)
• Audio: COVAREP features (74-dim, 500 timesteps)
• Video: OpenFace features (35-dim, 500 timesteps)