emotion2vec-web-distill

emotion2vec-web-distill (student v4)

A 2.23M-parameter, 9.7 MB ONNX distillation of emotion2vec+ large (~300M params, ~650 MB as fp32 ONNX) that runs in the browser in real time on plain WASM — no server, no WebGPU required. Raw 16 kHz waveform in, 18 emotion-anchor cosines out, with the mel frontend baked into the graph so the JavaScript side does zero DSP.

It was built to drive a game avatar's facial expression from live microphone audio. It is not a general speech-emotion-recognition system: it is a small, fast approximation of one teacher's opinion, published because a speech-emotion model this size that runs client-side did not seem to exist.

What it predicts

The teacher's classifier projects its embedding onto 9 emotion classes (angry, disgusted, fearful, happy, neutral, other, sad, surprised, unknown). We defined 18 expression "pose anchors" as unit-normalized weighted mixes of those classifier projection rows (weights in distill-anchor-mix.json — e.g. tense = 0.5·angry + 0.5·fearful, content = 0.5·happy + 0.5·neutral).

The training target for a 3-second window is the cosine similarity between the teacher's window embedding and each of the 18 anchors:

cos18 = normalize(teacher_embedding) @ anchors.T

The student learns to predict those 18 cosines directly from audio. Poses, in output order: calm, content, warm, happy, amused, excited, surprised, smug, confused, skeptical, annoyed, disgusted, afraid, angry, tense, sad, weary, bored.

This is a distillation of a model's judgment, not of human ground truth. The teacher's biases (and ours, via the anchor mixes) are inherited.

I/O contract

To turn cosines into blend weights, apply softmax(cosines / T) with the temperature from distill-student-v4.fused-meta.json.

Why 0.0375 and not the teacher's 0.05: distillation compresses the spread of the output cosines. Measured on 96,577 never-trained windows, the teacher/student deviation ratio is ~1.33× and uniform across all 18 anchors, so a single temperature correction (0.05 / 1.33) restores the teacher's weight distribution exactly, and — because argmax is temperature-invariant — without changing which emotion wins. The calibration is a property of this model generation; a future v5 will carry its own number in its own meta.

Usage (browser, onnxruntime-web)

import * as ort from 'onnxruntime-web';

const meta = await (await fetch('distill-student-v4.fused-meta.json')).json();
const session = await ort.InferenceSession.create('distill-student-v4.fused.onnx');

// Last 3 s of a 16 kHz mono mic ring buffer, float32 in [-1, 1].
async function emotionWeights(wave /* Float32Array(48000) */) {
  const { cosines } = await session.run({
    waveform: new ort.Tensor('float32', wave, [1, 48000]),
  });
  const T = meta.output.softmax_temperature;
  const z = Array.from(cosines.data, (c) => c / T);
  const m = Math.max(...z);
  const exps = z.map((v) => Math.exp(v - m));
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((v) => v / sum); // index-aligned with meta.output.poses
}

Tip: if you run this alongside another onnxruntime-web session on the WebGPU backend, serialize all session.run() calls through one queue — ort-web's WebGPU backend throws Session mismatch when two sessions' runs interleave.

Python sanity check:

import numpy as np, onnxruntime as ort
sess = ort.InferenceSession("distill-student-v4.fused.onnx")
wave = np.zeros((1, 48000), np.float32)  # substitute real 16 kHz audio
cos = sess.run(None, {"waveform": wave})[0]  # (1, 18)

Architecture and training

• Student: 64-mel log spectrogram → 4 conv stages (32→64→128→256) → 2-layer pre-norm transformer encoder (d=256, 4 heads, GELU) → mean-pool → linear head to 18 cosines. 2,231,986 parameters.
• Objective: MSE on standardized (z-space) targets; SpecAugment; early stopping on held-out z-MSE (best: epoch 18, z-MSE 0.5646).
• Splits: grouped so no speaker, actor, or meeting series straddles the train/held-out line. 351,281 train / 95,468 held-out windows.
• Fused export: the exact torchaudio mel frontend the student trained behind (n_fft 400, hop 160, 64 mels, 50–7600 Hz, HTK scale, log(·+1e-6)) is reproduced inside the ONNX graph as a conv1d STFT + mel matmul, and the input/output standardization constants are baked in. Export is parity-checked against the PyTorch model on real audio.

Evaluation

Ground truth is the teacher's window-level output. const predicts the train-mean cosines; ridge is a ridge regression from 128-d mel summary statistics. Top-anchor agreement = the student's argmax matches the teacher's.

Mean absolute error per face channel lands well inside the blend deadband of the target application; the numbers above are the stricter argmax view.

Read the baselines honestly. On calm-dominated splits (drama, read speech, spontaneous human speech) always answering "calm" is a strong baseline, and the student only ties it. The diagnostic that removes that flattery — windows where the teacher says something other than calm:

Known weaknesses: on real spontaneous speech the student is far more conservative than its teacher — when the teacher hears a non-calm emotion in free speech, the student agrees only ~14% of the time and rarely produces angry at all. Distinct anger-placement work is planned for v5. For its built purpose (subtle continuous expression blending, where the temperature calibration restores full dynamic range) this conservatism is acceptable; for anything that depends on catching emotional events in natural conversation, it is not.

Training data

Targets are the teacher's outputs over these corpora; no human emotion labels were used for training (CREMA-D's nominal labels served as diagnostics only).

Privacy: no private voice data was used in training. The maintainer's own recordings were used exclusively for evaluation ("human" rows above) and are not distributed.

Intended use and limitations

Intended: driving continuous, low-stakes expressive output — avatar faces, ambient visualization, creative tools — client-side, from a user's own microphone, with the audio never leaving their machine.

Not intended, and not validated for: emotion analytics on other people, hiring or admissions screening, surveillance, clinical or mental-health assessment, lie detection, or any consequential decision about a person. Beyond the ethics, the numbers above say it plainly: this model misses most non-calm emotional events in natural speech. It predicts what one model would say about a 3-second window of mostly-English audio; it does not read minds.

License and attribution

The teacher, emotion2vec/emotion2vec_plus_large, is released under the FunASR Model Open Source License, which covers "model weights and their derivatives, including finetuned models" and requires sharing under the same terms with attribution. This distilled student is such a derivative (trained on the teacher's outputs; its output space is built from the teacher's classifier rows), so it is published under the same FunASR Model License (see LICENSE). Note the license does not explicitly address commercial use and states the models are provided "for reference and learning purposes"; evaluate it for your own use case.

Credit for everything the student knows goes to the emotion2vec authors and the FunASR team at Alibaba:

@article{ma2023emotion2vec,
  title={emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation},
  author={Ma, Ziyang and Zheng, Zhisheng and Ye, Jiaxin and Li, Jinchao and Gao, Zhifu and Zhang, Shiliang and Chen, Xie},
  journal={arXiv preprint arXiv:2312.15185},
  year={2023}
}
@inproceedings{gao2023funasr,
  title={FunASR: A Fundamental End-to-End Speech Recognition Toolkit},
  author={Gao, Zhifu and Li, Zerui and Wang, Jiaming and Luo, Haoneng and Shi, Xian and Chen, Mengzhe and Li, Yabin and Zuo, Lingyun and Du, Zhihao and Xiao, Zhangyu and Zhang, Shiliang},
  booktitle={INTERSPEECH},
  year={2023}
}

Release source

This model payload is published automatically from the validated publish/ directory on the main branch of the source repository. The model metadata's SHA-256 identifies the exact ONNX payload independently of repository history.