michi-audio-affect

Michi Audio Affect

An English speech-emotion classifier built for the fast listener in Michi. It classifies 16 kHz mono speech into six coarse vocal-tone labels:

The repository contains the quantized ONNX runtime artifact. It does not contain training audio.

Intended use

This model is intended as a low-latency, secondary signal for conversational presentation. Michi combines its output with text emotion and deterministic rules; the model never changes plans, prices, reservations, or other application state.

Do not use it to diagnose mental health, infer stable personal traits, make high-impact decisions, or treat a predicted emotion as a fact about a person.

Training

• Base encoder: facebook/wav2vec2-base, frozen at revision 0b5b8e8
• Dataset: myleslinder/crema-d, revision 8a11ae8
• Input: 16 kHz speech audio
• Feature: mean-pooled hidden state from encoder layer 7
• Head: multinomial logistic regression
• Split: actor-disjoint; eight actors held out for validation and the two local fixture actors excluded from training
• Export: Wav2Vec2ForSequenceClassification → ONNX → dynamic int8 quantization

The dependency- and revision-pinned build recipe is in scripts/train-audio-affect-head.py in the Michi repository. It is intended to be repeatable, but does not promise bit-for-bit identical artifacts across operating systems and hardware.

Validation

The logistic-regression head reached 78.2% accuracy across six classes on an actor-disjoint validation split. The same validation split selected the best of 13 encoder layers, so this is a model-selection result rather than an estimate from an untouched test set. The score was calculated before ONNX export; the quantized artifact has runtime behavior checks, but has not been evaluated over the complete validation split.

The pre-export confusion matrix, validation actor IDs, and source revisions are in TRAINING_REPORT.json.

Usage with Transformers.js

import { pipeline } from "@huggingface/transformers";

const classify = await pipeline(
  "audio-classification",
  "cris-cmd/michi-audio-affect",
  { dtype: "q8" },
);

// Float32Array containing mono, 16 kHz PCM samples.
const scores = await classify(audio, { top_k: null });

Michi pins a downloaded copy under models/affect/michi-audio-affect so it can run without a model-host request during a conversation.

Limitations

• Trained on acted English speech using a small set of fixed sentences.
• The six labels are intentionally coarse and do not cover natural emotional complexity.
• Accuracy will vary with language, accent, microphone quality, background noise, and spontaneous speech.
• Treat scores as uncertain presentation hints and combine them with text and explicit user intent.

Licenses and attribution

The exported model is released under Apache-2.0. The base encoder is Apache-2.0. CREMA-D is distributed under the Open Database License, with individual contents covered by the Database Contents License. Users are responsible for complying with the source dataset terms.

CREMA-D citation:

@article{cao2014crema,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G. and Keutmann, Michael K. and Gur, Ruben C. and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  volume={5},
  number={4},
  pages={377--390},
  year={2014},
  doi={10.1109/TAFFC.2014.2336244}
}