averformer-causal-av-cremad

Live causal audio-visual emotion recognition

A causal audio-visual emotion recognizer: it predicts from past audio and video only, never future frames, so the same model runs offline on a clip and live on a webcam at 4 predictions/second. Recognises 6 emotions: angry, disgust, fear, happy, neutral, sad.

Everything needed to run is in this repo — it works fully offline after download.

Accuracy — measured, please read

CREMA-D fold01 test split: 18 speakers the model never trained on, near class-balanced 6-class, chance = 0.167. Selected on a held-out validation split (never on test).

("WAR" in the source repo means weighted average recall = overall accuracy, not balanced accuracy; the two nearly coincide here only because this test split is near-balanced.)

Causality costs about 4 points against the teacher that gets to see the future.

Audio matters. Measured on a 72-clip slice of the same test split:

Turning the microphone off costs ~12 points; sad suffers most (0.58 → 0.25) — it is the class the voice carries. The 72-clip slice is small, so treat those three as indicative.

Limitations — what this model is not

• Trained on acted emotion. CREMA-D is actors performing an emotion on command. This recognises acted expression, which is not the same thing as what a person feels. Do not use it to make judgements about real people, and especially not consequential ones.
• Never evaluated on webcam video. Every number here is CREMA-D clips: studio lighting, frontal faces. Your camera, lighting and framing are out of distribution — expect worse.
• Six classes. Anything else you feel gets forced into the nearest of these six.
• Not state of the art. CREMA-D SOTA is ~0.895 with a non-causal model.
• English-speaking North American adult actors. Performance across other demographics is unmeasured, and face models are known to vary across skin tone and age.
• The live-camera tab is video-only (a browser frame stream has no synchronised mic), so it runs at the 0.68 row above. The Record clip tab uses camera + microphone together and is the path the 0.787/0.81 numbers refer to.

Run it locally (fully offline)

git lfs install
git clone https://huggingface.co/mhussainahmad/averformer-causal-av-cremad
cd averformer-causal-av-cremad
pip install -r requirements.txt        # plus ffmpeg from your system package manager

python run_local.py --check            # verify the offline install
python run_local.py --video clip.mp4   # analyse a video file
python app.py                          # the full live camera + mic UI at localhost:7860

No network is used at any point after the clone: the checkpoint carries the fine-tuned backbones, the encoder architectures are built locally, and the face detector is bundled. Verified with HF_HUB_OFFLINE=1 and all HTTP proxies pointed at a dead port.

How it works

Causality is structural, not a training trick: the visual encoder sees one frame, the audio encoder one trailing window, and the temporal head only shifts past→present. The clip contract is 16 frames over 4 s → a 0.25 s hop (4 Hz) with a 0.5 s trailing audio window. Streaming encodes only the newest frame and window, reusing cached features for the rest.

weights/model_cremad_f01_s2024.pth (113 MB) is the whole model: both fine-tuned backbones plus the head, 28.2 M parameters.

Training

CREMA-D fold01, seed 2024. Frozen-then-fine-tuned pretrained encoders, cross-entropy only, subject-disjoint validation split, weight EMA, epoch selected on validation accuracy.

There is no knowledge distillation in this model, despite the "distill" naming in the source repo — it was tried and it hurt (0.294 vs 0.42). Pretrained-encoder initialization is the entire lever. Reported as measured, not as designed.

Labels

Emitted in this order, verified by joining each CREMA-D filename's emotion code to its integer label across every row of every training fold (zero conflicts):

0 angry   1 disgust   2 fear   3 happy   4 neutral   5 sad

License & credits

MIT for this code. CREMA-D is released under the Open Database License. HSEmotion (Savchenko), DistilHuBERT (NTU SPML), YuNet (OpenCV Zoo) retain their own licenses.