mouth-gate-nano

mouth-gate-nano

A 47,410-parameter visual-only "is this face speaking" model that did NOT beat its scalar baseline, published so the baseline and the numbers are citable and so it can be A/B-tested on real meeting faces. The recommendation for deployment is the scalar, not this model.

Domain measured / deployment domain tested: measured on AVA-ActiveSpeaker (movies; 12 train, 4 held-out test titles chosen for multi-face density; 480p; movie-split). Deployment domain (frontal webcam meeting faces) NOT tested; it may be easier or harder than movies.

Effective sample size (added 2026-09-05). The 6,105 test stacks come from 4 movies. Intraclass correlation by movie is 0.20, giving an effective sample size of about 20 (nanolab.effective_n); by face track it is 0.81 with n_eff ≈ 1,900. Read every number above as a four-movie result with a spread to match, not a six-thousand-sample one.

Measured, held-out movies, frames with 2+ faces, positive = audibly speaking

Baseline + margin. Mean absolute frame difference inside the face box over the 5-frame stack. The model beats it by +0.06 to +0.07 AP; our ship gate is +0.15. Ship the scalar.

Named failure. The lower third of an AVA head box is mostly chin (verified on a sample grid), which is why the "mouth" crop is worse than the whole face. Both crops are near chance on faces under ~100 px. 12 fps beats 4 fps in every row: at 4 fps five frames span 1.25 s and measure "did the mouth change", not lip kinematics.

Refusal. No face box -> no output (the composed system falls back to audio only). Faces under 64 px -> do not call.

Provenance. AVA-ActiveSpeaker v1.0 annotations (CC BY 4.0), 15-minute annotated windows fetched from the CVDF mirror, re-encoded 480p/25 fps. Visual only -- the model has no audio input by construction, so it cannot learn the audio envelope. Scripts mouth_stack_prep.py, mouth_gate_train.py, nano_to_tfjs.py in the loglens repo. Trained 2026-09-02, Jetson AGX Orin.

Files and input contract

• tfjs/model.json + tfjs/group1-shard1of1.bin — TensorFlow.js GraphModel, 201 KB, ops: Pad, _FusedConv2D, Mean, _FusedMatMul, AddV2, Mul (all in the WebGL backend). Parity with the PyTorch weights verified to 1e-6 relative.
• model.pt — PyTorch state_dict of nanolab.Nano(nout=2, cin=5).

Input: tensor [1, 64, 64, 5] NHWC float32. Five grey face crops (whole BlazeFace box, resized to 64x64 with area interpolation), oldest first, sampled ~12 fps, scaled to 0-1, then the whole stack standardised: (x - mean(x)) / (std(x) + 1e-6) over all 5x64x64 values. Output: logits [1, 2]; softmax[1] is P(speaking). Calibrate the threshold on your own faces; on AVA the precision-recall curve is flat, which is the point of this card.