wav2vec2-msp-dim-emotion-coreml

wav2vec2 MSP-Dim Emotion — Core ML conversion

A Core ML (.mlpackage) conversion of audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim for on-device dimensional speech emotion recognition (arousal / dominance / valence) on Apple Silicon (Neural Engine / GPU / CPU via MLComputeUnits.all).

⚠️ Research / non-commercial use only. This is a derivative work and inherits the original model's CC BY-NC-SA 4.0 license. See License & attribution below.

⚖️ License & attribution (please read)

This model is a derivative work of audEERING's model and is released under the same license: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC BY-NC-SA 4.0).

• Original model: © audEERING GmbH — audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim ("for research purpose only"). A commercial license for the original can be obtained from audEERING.
• Base architecture: Meta AI — Wav2Vec2-Large-Robust.
• Training data: MSP-Podcast (v1.7), The University of Texas at Dallas — research use.
• Changes made in this derivative: exported the PyTorch model to a Core ML ML Program (FLOAT32) via coremltools; the Wav2Vec2 feature normalization (zero-mean/unit-variance) is baked into the graph, so the model consumes a raw waveform directly. No weights were retrained or altered.
• NonCommercial: do not use this model (or works built on it) commercially without a commercial license from audEERING. ShareAlike: redistribute only under CC BY-NC-SA 4.0.
• Not affiliated with or endorsed by audEERING, Meta, or UT Dallas.

Citation

@article{wagner2023dawn,
  title   = {Dawn of the Transformer Era in Speech Emotion Recognition:
             Closing the Valence Gap},
  author  = {Wagner, Johannes and Triantafyllopoulos, Andreas and Wierstorf, Hagen and
             Schmitt, Maximilian and Burkhardt, Felix and Eyben, Florian and Schuller, Bj{\"o}rn W.},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year    = {2023},
  note    = {arXiv:2203.07378}
}

Outputs

A single tensor of 3 values in the order [arousal, dominance, valence], each ≈ [0, 1]. Valence is index 2. To map valence to a bipolar [-1, 1] scale: clamp(2·v − 1, −1, 1).

Input contract

• Raw float32 mono waveform at 16 kHz, shape [1, T].
• Supported length 0.3 s – 30 s (4800 – 480000 samples).
• Do not normalize — normalization is baked into the graph; pass raw samples.

Conversion fidelity

Validated numerically identical to the PyTorch reference (FLOAT32): Pearson r = 1.000000, mean/max abs error ≈ 0 on valence, arousal, and dominance across the validation set.

Usage (Swift / Core ML)

import CoreML

let cfg = MLModelConfiguration()
cfg.computeUnits = .all
let compiled = try await MLModel.compileModel(at: mlpackageURL)
let model = try MLModel(contentsOf: compiled, configuration: cfg)

// samples: [Float] raw mono 16 kHz, 0.3–30 s
let x = try MLMultiArray(shape: [1, NSNumber(value: samples.count)], dataType: .float32)
for (i, s) in samples.enumerated() { x[i] = NSNumber(value: s) }
let out = try model.prediction(from: try MLDictionaryFeatureProvider(dictionary: ["input_waveform": x]))
// read the 3-vector output; valence = index 2; bipolar = clamp(2v-1, -1, 1)

Intended use

On-device, privacy-preserving prosody/affect analysis of meeting audio (research use), e.g. deriving a per-speaker acoustic valence to compare against text sentiment ("tone vs. words"). Dimensional SER models carry speaker/timbre and dataset biases — prefer within-speaker trends over cross-speaker absolute comparisons, and never treat a single score as ground truth about a person's emotional state.