korean-ser-wav2vec2-xlsr-4class

Korean Speech Emotion Recognition: wav2vec2-XLSR (4-class)

facebook/wav2vec2-large-xlsr-53 fine-tuned for Korean speech emotion recognition. Takes a waveform, returns one of four emotions. Single model, no ASR, no text input.

Evaluated on a speaker- and script-disjoint test split of 2,451 utterances.

Usage

The checkpoint loads through the framework it was trained with:

git clone https://github.com/furence-ai/Korean-SER-Benchmark
cd Korean-SER-Benchmark && uv sync

uv run python -m scripts.infer --ckpt korean_ser_wav2vec2_xlsr_4class.pt sample.wav
# → sample.wav → happy (conf=0.984) | happy=0.984 sad=0.007 angry=0.004 neutral=0.006

From Python:

import torch
from ser.engine import build_model_from_ckpt
from ser.audio import load_audio_16k_mono

ckpt  = torch.load("korean_ser_wav2vec2_xlsr_4class.pt", map_location="cpu", weights_only=False)
model = build_model_from_ckpt(ckpt, "cpu")            # returns an eval-mode nn.Module

wav = load_audio_16k_mono("sample.wav").unsqueeze(0)  # (1, T) float32, 16 kHz mono
with torch.no_grad():
    probs = torch.softmax(model(wav), dim=-1)[0]

names = ckpt["label_to_english"]                      # {0:'happy',1:'sad',2:'angry',3:'neutral'}
print({names[i]: round(float(probs[i]), 4) for i in range(ckpt["num_classes"])})

Audio must be 16 kHz mono. Training cropped clips at 10 seconds; longer input is truncated the same way.

Performance

Per-class F1 on the held-out test split:

The confusion matrix is included as confusion_matrix.png.

For context, this model scores 0.8588 where the same backbone with a frozen encoder reaches 0.5999 and the off-the-shelf emotion2vec baseline reaches 0.5877 on the identical split. Adding a text branch gets to 0.8880 but requires a transcript; see Optional: text fusion.

The full 19-configuration leaderboard, confusion matrices and reproduction instructions are in the GitHub repository.

About the evaluation split

The training corpus pairs each of its 9,182 scripts 1:1 with an emotion, and each script is read by roughly 32 speakers. Holding out speakers alone therefore leaves the test sentences inside the training set, which inflates results. This model was trained and evaluated on a split where speakers and scripts are both disjoint, so its numbers are not directly comparable to higher ones reported on the same corpus under a looser split.

Optional: text fusion

A separate 0.6 M-parameter head combines this model's embeddings with KLUE-RoBERTa text embeddings for 0.8880 macro-F1 (+2.9 points). It needs a transcript at inference and carries a different license. See furence-ai/korean-ser-wav2vec2-xlsr-4class-fusion-head.

Unless you already have transcripts, this model alone is the better trade: 97% of the fusion score from one model instead of three.

Training

License

Apache-2.0, derived from facebook/wav2vec2-large-xlsr-53 (Apache-2.0). No share-alike or non-commercial dependency applies to this model.

Training data: the 감성 및 발화 스타일별 음성합성 데이터 corpus (AI-Hub; built under the Ministry of Science and ICT / NIA "Intelligent Information Industry Infrastructure" programme) is licensed per approved user and may not be redistributed. Neither this repository nor the GitHub repository contains any audio, manifest or transcript, only model weights and evaluation results. To reproduce training, obtain your own approved copy from AI-Hub.

Citation

@misc{korean_ser_wav2vec2_xlsr,
  title  = {Korean Speech Emotion Recognition: wav2vec2-XLSR fine-tuned on a leakage-free split},
  year   = {2026},
  note   = {https://github.com/furence-ai/Korean-SER-Benchmark}
}