Downloads · 30 days
0
furence-ai/korean-ser-wav2vec2-xlsr-4class-fusion-head
korean-ser-wav2vec2-xlsr-4class-fusion-head is a audio classification model from furence-ai. Use it for the audio classification task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as other.
⚠️ This is not a standalone model. It is a 0.6 M-parameter MLP over concat[acoustic 1024, text 1024]. On its own it does nothing. You need the acoustic encoder, KLUE-RoBERTa, and a transcript.
Downloads · 30 days
0
Access
Public
Updated Aug 19, 2026
Repo size
2.4 MB
Likes
0
Public
Click a slice to open those files.
.pt2.4 MB · 99%
From the Hugging Face model README
⚠️ This is not a standalone model. It is a 0.6 M-parameter MLP over
concat[acoustic 1024, text 1024]. On its own it does nothing. You need the acoustic encoder, KLUE-RoBERTa, and a transcript.
Combines acoustic and text embeddings for Korean speech emotion recognition. Adds +2.9 macro-F1 over the acoustic model alone, at the cost of a transcript and two extra models at inference.
| Task | 4-class speech emotion recognition (Korean) |
| Labels | 0 happy · 1 sad · 2 angry · 3 neutral |
| Input | concat[acoustic 1024, text 1024] → 2048-d vector |
| Parameters | 0.6 M (head only) |
| macro-F1 | 0.8880 (accuracy 0.8947) |
| Component | Where |
|---|---|
| Acoustic encoder | furence-ai/korean-ser-wav2vec2-xlsr-4class |
| Text encoder | klue/roberta-large (frozen) |
| Transcript | you supply it (no ASR is bundled) |
| Code | GitHub repository |
The provenance is recorded inside the checkpoint itself (acoustic_backbone,
text_backbone, acoustic_ckpt).
git clone https://github.com/furence-ai/Korean-SER-Benchmark
cd Korean-SER-Benchmark && uv sync
uv run python -m scripts.infer_fusion --method early \
--acoustic-ckpt korean_ser_wav2vec2_xlsr_4class.pt \
--fusion-ckpt korean_ser_wav2vec2_xlsr_4class_fusion_head.pt \
sample.wav --text "오늘 정말 기분이 좋아"
Pass whatever STT you already run as --text. The repository deliberately
bundles no ASR, so the text branch has no fixed upstream dependency.
| Configuration | macro-F1 | Models to run |
|---|---|---|
| early fusion (this head) | 0.8880 | 3 |
| cross-attention fusion | 0.8858 | 3 |
| late fusion (logit averaging) | 0.8763 | 3 |
| acoustic only | 0.8588 | 1 |
| text only (KLUE-RoBERTa) | 0.5961 | 1 |
All rows share one speaker- and script-disjoint split, one engine and one
metric. The confusion matrix is included as confusion_matrix.png; the full
19-configuration leaderboard is in the
GitHub repository.
Is it worth it? The gain is real but small. Plain concatenation beats the much more expensive cross-attention variant, and text alone reaches only 0.5961. If you do not already have transcripts, the acoustic model delivers 97% of this score with one model and no ASR.
| Data | 감성 및 발화 스타일별 음성합성 데이터 (AI-Hub) |
| Split | speaker- and script-disjoint, seed 42 (190,837 train / 2,933 val / 2,451 test) |
| Acoustic embeddings | fine-tuned wav2vec2-XLSR, frozen, pre-extracted |
| Text embeddings | klue/roberta-large, frozen, mean-pooled, pre-extracted |
| Head | MLP over the 2048-d concatenation (hidden 256, dropout 0.2) |
| Optimiser | AdamW, LR 5e-4 |
| Selection | best validation macro-F1, early stopping patience 5 (stopped at epoch 6) |
Both encoders are frozen; only the head is trained, on cached embeddings.
⚠️ Different from the acoustic model, which is clean Apache-2.0.
This head was trained on embeddings produced by
klue/roberta-large, which is
released under CC BY-SA 4.0 (share-alike). Whether that obligation attaches
to a model trained on its outputs is not clearly settled, so this repository is
marked other rather than asserting either answer. Review the share-alike
terms before redistributing this head or anything derived from it, and take
your own legal advice.
If you need an unambiguous license, use the acoustic model instead. It is Apache-2.0 end to end, with no share-alike dependency.
Training data: the 감성 및 발화 스타일별 음성합성 데이터 corpus (AI-Hub; built under the Ministry of Science and ICT / NIA "Intelligent Information Industry Infrastructure" programme) is licensed per approved user and may not be redistributed. No audio, manifest or transcript is included here.
@misc{korean_ser_early_fusion,
title = {Korean Speech Emotion Recognition: early fusion of wav2vec2-XLSR and KLUE-RoBERTa},
year = {2026},
note = {https://github.com/furence-ai/Korean-SER-Benchmark}
}