emotion_to_mesh

emotion + pose → MediaPipe 478 mesh PLS

Current default: emotion_to_mesh_pls_v6.npz (Detectorv2 v2.8 emotion space; py-feat load_emotion_face_mesh_model() / plot_face_mesh(emotion=...)).

emotion_to_mesh_pls_v5.npz is deprecated — its column labels are permuted. Column j of v5 holds the network's j-th output (Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger) but is labelled in FEAT_EMOTION_COLUMNS order (anger, disgust, fear, happiness, sadness, surprise, neutral). Indexing v5 by its labels renders the wrong expression (e.g. "anger" renders neutral). The file is retained unchanged for reproducibility only.

Versions

Column order

feature_columns = feat.utils.FEAT_EMOTION_COLUMNS (anger, disgust, fear, happiness, sadness, surprise, neutral). Detectorv2's Fex emits Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger; py-feat matches dict / Series / DataFrame inputs by name and accepts either naming. Raw arrays must be in feature_columns order.

Training data (v6)

• 602,427 frames / 33,775 CelebV-HQ videos run through the py-feat Detectorv2 v2.8 release weights (face_multitask_v28.safetensors, md5 9e174bb3cd05ce8bed7ac64358a40332). Features and target meshes come from the same forward pass, so the map is self-consistent with what the detector emits.
• Same frames, pose filter (|yaw| ≤ 40°, |pitch| ≤ 30°), 12-anchor GPA alignment and top-1% anchor-residual outlier drop as py-feat/au_to_mesh v6.
• Canonicalized with an Umeyama similarity on all 478 vertices onto the au_to_mesh v4 frontal mean, the same recipe as au_to_mesh_pls_v6, so AU, emotion and blendshape renders share one frame.

Method (v6)

• Training design [7 emotion | 3 pose | 21 pose × emotion] = 31 features; kernel PLS equivalent to PLSRegression(scale=True), k chosen by mean held-out R² (k = 27; knee at 16).
• Deployed rows = emotion + pose (interactions vanish at pose = 0).
• mean_neutral_mesh = prediction at one-hot neutral.
• Per-fold R² = [0.3115, 0.3064, 0.2939, 0.3049, 0.3103].

Behaviour and limitations

Relative to one-hot neutral: happiness raises and widens the lip corners; sadness lowers the corners and raises the brows; surprise raises the brows and opens the jaw; fear widens the eyes; anger lowers the brows and narrows the inner-brow gap. All non-neutral emotions also open the mouth somewhat, reflecting the speech-heavy CelebV-HQ frames on which emotional expressions were observed. R² is modest because a 7-way softmax carries little geometric information; use this model for qualitative visualization.

Inference

from feat.plotting import plot_face_mesh, predict_face_mesh_from_features
plot_face_mesh(emotion={"Happy": 1.0})                  # Detectorv2 or py-feat names

Or with numpy only:

import numpy as np
m = np.load("emotion_to_mesh_pls_v6.npz")
cols = m["feature_columns"].tolist()
x = np.zeros(len(cols) + 3, dtype=np.float32)     # [features | Pitch, Yaw, Roll]
x[cols.index(...)] = ...
flat = x @ m["coef"] + m["intercept"]             # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)  # axis-major [x|y|z]

File format

NPZ: coef (k+3, 1434) f32; intercept (1434,) f32; input_columns; feature_columns (authoritative order; index by name); feature_name; pose_columns; mean_aligned_mesh (478, 3); mean_neutral_mesh (478, 3); reference_anchors (12, 3); anchor_indices (12,); n_components; model_card; training_metadata (JSON).

License

MIT for this artifact. The upstream detector that generated its training data carries research-only restrictions.