au_to_mesh

AU → MP-mesh PLS (visualization)

Predicts the 478-vertex MediaPipe FaceMesh deformation from 20 AU intensities. Trained with pose covariates and pose×AU interactions for cleaner AU coefficients; deployed as a 23-d input (AU + pose), with pose=0 at inference for the standard canonical-frontal visualization.

Backs py-feat's plot_face_mesh(au=...) / predict_face_mesh — the AU→geometry inverse of the detector.

Current default: au_to_mesh_pls_v6.npz.

Versions

Match the PLS version to your detector. Each model is fit on the AU distribution of a specific detector; pairing v6 with a pre-2.8 detector (or an older PLS with 2.8) maps AU values through a distribution the model was not fit on. py-feat ≥ 2.1.3 ships v2.8 weights and defaults to v6, so the pairing is correct out of the box. All older versions remain selectable via load_face_mesh_viz_model(model_version=...).

v6 training data

• 672,970 frames / 35,664 CelebV-HQ videos run through the v2.8 release checkpoint (soupGen, the mean of stage-3 epochs 2/6/19 — the same weights shipped as face_multitask_v28.safetensors).
• Targets are the detector's own predicted 478-vertex mesh, so the mapping is self-consistent with the AU space it is driven by.
• Pose-filtered to |yaw| ≤ 40°, |pitch| ≤ 30° → 608,513 frames; top 1% by max anchor residual dropped as alignment outliers → 602,427 frames / 33,920 videos fit.
• Per-frame Umeyama similarity Procrustes + iterative GPA to a population-mean template, on 12 stable anchors (forehead 10/9/8/151, nose bridge 6/168/197/195, outer canthi 33/263, inner canthi 133/362). Removes (R, s, t).
• No per-subject neutral subtraction — absolute aligned coordinates, the Cheong / py-feat-tutorial-06 recipe.

v6 method

• PLSRegression(n_components=83, scale=True).
• Inputs during fitting: [20 AU | 3 pose | 60 pose×AU] = 83. Only the 23-row AU+pose slice is deployed; pose is zero at inference, so the interaction terms vanish and the shipped model is a pure AU→shape map.
• Linearized to an explicit (coef, intercept) on the identity basis (linearization error 1.3e-15), then rigidly mapped into the v4 canonical frontal frame so every version renders at the same orientation and scale.

v6 performance

Variance-weighted R² = 0.4725 ± 0.0045, 5-fold GroupKFold by video (no video appears in both train and test), across all 1434 output dims. MAE = 0.0047 in aligned units.

Full k sweep (PLS components are nested, so one fit per fold yields the whole curve):

Against v5 at its own published grid points: 0.4505 vs 0.347 (k=20), 0.4656 vs 0.358 (k=40), 0.4706 vs 0.361 (k=60), 0.4725 vs 0.362 (k=83).

Fold count is immaterial at this sample size — 3-fold and 5-fold agree to ≤0.0005 across the entire curve.

Two changes contribute to the gain over v5 and are not separately identified: a better-conditioned v2.8 AU space, and a corrected extraction path (v5's inputs were center-cropped 256→224, while v2.5+ models and py-feat's shipped inference both resize — worth 0.05 mean absolute AU error and 4.6° of head pose).

R² is modest in absolute terms because 20 AU intensities cannot fully describe a 1434-d mesh deformation. For visualization, qualitative AU-direction correctness matters more than R²; see Known limitations for the AUs whose v6 deformation is not anatomically specific.

Known limitations

Rendered at AU = 1.5 (beyond the input range, to make small effects visible):

• Clean, localized: AU01, AU02, AU04, AU05, AU06, AU12, AU15, AU25, AU26, AU43.
• Opens the mouth / jaw: AU20 (lip stretcher) and AU09 (nose wrinkler) carry jaw opening that is not part of the FACS definition.
• Broad lower-face motion: AU07, AU11, AU14, AU17 and AU24 move most of the lower face rather than a specific region; AU17 differs most from v5 (coefficient cosine 0.27).

These reflect AU co-activation in natural video, inherited by a linear map on absolute coordinates, rather than a packaging error. Companion models for emotion (py-feat/emotion_to_mesh) and blendshapes (py-feat/bs_to_mesh) v6 share this model's frame.

Inference

import numpy as np
m = np.load("au_to_mesh_pls_v6.npz")
au = np.zeros(20); au[m["au_columns"].tolist().index("AU12")] = 1.0  # smile
pose = np.zeros(3)                                                   # [pitch, yaw, roll]
x = np.concatenate([au, pose])                                       # (23,)
flat = x @ m["coef"] + m["intercept"]                                # (1434,)
# IMPORTANT: layout is axis-major [all x | all y | all z], NOT interleaved
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)     # (478, 3)
# Render with mediapipe.solutions.face_mesh.FACEMESH_TESSELATION

Or through py-feat:

from feat.plotting import plot_face_mesh, predict_face_mesh
plot_face_mesh(au={"AU12": 1.0})          # defaults to v6

AU inputs are probabilities in [0, 1]. Values above 1.0 are outside the model's input domain (the detector never emits them); because the model is linear, driving past 1.0 only scales the deformation and adds no information.

predict(au=0) equals mean_aligned_mesh by construction, so the "no AU given" neutral render and the AU=0 prediction agree.

Applying pose post-hoc (optional)

The model is trained with pose covariates but at inference users typically pass pose=0 for the canonical frontal deformation. To render at a chosen head pose, apply a rigid transform after prediction:

from scipy.spatial.transform import Rotation
R = Rotation.from_euler("xyz", [pitch, yaw, roll]).as_matrix()   # (3, 3)
posed_mesh = mesh @ R.T
# Or for re-projection onto an image: s * (mesh @ R.T) + t

Two ways to control pose:

• Render-time rotation (recommended): set pose=0, rotate post-hoc as above. The AU-driven deformation stays canonical and pose is purely a viewing transform.
• Pose-conditioned prediction: pass non-zero pose into the input vector to get the model's residual pose-correlated bias in addition to the rigid rotation needed at render time.

Convention notes:

• MP canonical frame: y-axis up, x-axis to the subject's left, z-axis out of the face. Right-handed.
• Rotation.from_euler("xyz", ...) is intrinsic xyz (R = Rx·Ry·Rz).
• With MP's 4×4 facial_transformation_matrix (M): posed = (np.concatenate([mesh, np.ones((478, 1))], axis=1) @ M.T)[:, :3]

File format

NPZ with:

coef               (23, 1434) float32  linear weights, rows match input_columns
intercept          (1434,)    float32  bias (== mean_aligned_mesh, flattened)
input_columns      (23,)      str      AU01..AU43, Pitch, Yaw, Roll
au_columns         (20,)      str      the AU subset
pose_columns       (3,)       str      the pose subset
mean_aligned_mesh  (478, 3)   float32  population mean canonical mesh ("AU=0" face)
reference_anchors  (12, 3)    float32  Procrustes reference
anchor_indices     (12,)      int32    MP indices used as anchors
n_components       ()         int32
model_card         ()         str

Loader: np.load("au_to_mesh_pls_v6.npz") — no extra dependencies.

Note v2 additionally carried mean_low_au_mesh and training_metadata; v4+ do not.

Figures (v6)

Each AU driven to 1.0, its full in-range maximum.

The au_solid_*.png and au_compare_overlay.png files in this repo are from the original v2 model and are retained for history only.

Citation context

Adapts Cheong et al. 2023 / py-feat tutorial 06 (E. Jolly): affine-aligned 68 dlib landmarks + pose → 20 AUs via PLS. Direction inverted (AU+pose → mesh) and scaled to MediaPipe's 478-vertex mesh on wild-celebrity video.

License

MIT for this artifact. Note the upstream detector that generated its training targets carries research-only restrictions.