face-capture-quality

Face Capture Quality (FCQ)

Pose-aware face-capture-quality + head-pose model. Given a detected face crop it predicts head pose (yaw/pitch/roll in degrees) and a pose-independent usability score, plus blur/occlusion/exposure flags. Built to gate guided multi-angle ("360") face capture, where off-the-shelf quality scores (e.g. Apple's VNDetectFaceCaptureQuality) wrongly penalise non-frontal poses: a sharp, well-lit three-quarter turn should score high, a blurry frontal face should score low.

• Backbone: MobileNetV3-Small (timm mobilenetv3_small_100), ~1.9M params.
• Output (single tensor, 7): [yaw, pitch, roll, usable, blur, occluded, bad_exposure].
• Formats: pytorch_model.bin, model.onnx (opset 17), FaceCaptureQuality.mlpackage (CoreML, fp16).

Input / preprocessing

RGB, NCHW, 160x160, values in [0, 1] (ImageNet normalisation is baked into the graph; CoreML uses ImageType(scale=1/255)). The crop is a square, axis-aligned bounding-box crop around the detected face (~0.2 margin), resized to 160 — i.e. pose-preserving, NOT eye-aligned (a head-pose model must keep roll/yaw/pitch). The host app should feed the same: detect face -> square bbox + ~20% margin -> resize 160.

How to use (ONNX Runtime)

The model expects a face crop — detect the face first, take a square bounding box with ~20% margin, then resize to 160×160 RGB in [0, 1], NCHW. ImageNet normalisation is baked into the graph, so no manual mean/std subtraction is needed.

import numpy as np, onnxruntime as ort
from PIL import Image
from huggingface_hub import hf_hub_download

onnx_path = hf_hub_download("shafi-afridi/face-capture-quality", "model.onnx")

# face_crop.jpg = an already-detected face (square bbox + ~20% margin)
face = Image.open("face_crop.jpg").convert("RGB").resize((160, 160))
x = (np.asarray(face, np.float32) / 255.0).transpose(2, 0, 1)[None]  # (1, 3, 160, 160)

sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
out = sess.run(None, {sess.get_inputs()[0].name: x})
yaw, pitch, roll, usable, blur, occluded, bad_exposure = np.asarray(out).reshape(-1)[:7]

print(f"pose°:  yaw={yaw:.1f}  pitch={pitch:.1f}  roll={roll:.1f}")
print(f"usable={usable:.2f}  blur={blur:.2f}  occluded={occluded:.2f}  bad_exposure={bad_exposure:.2f}")
# yaw/pitch/roll in degrees; usable + flags in [0, 1] (sigmoid). Higher `usable` = better capture.

On Apple devices prefer CoreML: download FaceCaptureQuality.mlpackage and feed a Vision/Core ML ImageType with scale=1/255 (normalisation is already inside the graph).

Intended use

On-device capture guidance (CoreML / ONNX). NOT a recognition or identity model, and not an authentication or surveillance tool.

Results

Numbers are produced by eval.py; see the GitHub repo to reproduce. The ERC injects controlled degradations into LFW to create a quality range, then shows that discarding the lowest-usable faces lowers verification error.

Training data & licence

Head pose is trained on 300W-LP (real ground-truth pose Pose_Para; pose-diverse via synthetic large-pose renders) and evaluated on AFLW2000-3D — the standard head-pose protocol, cropped identically (square box around the 68 landmarks, no detector). The quality head is trained on synthetic degradations (controlled blur / JPEG / noise / exposure / occlusion with known severity) applied to 300W-LP + FFHQ faces — FFHQ supplies real, high-resolution clean faces so the model calibrates "crisp real face = high usable" — plus real occlusion/blur/framing negatives from WIDER FACE.

300W-LP, FFHQ, WIDER FACE and AFLW2000-3D are research / non-commercial datasets, so the v1 weights are released cc-by-nc-4.0 — research use only. A commercially-licensed version would require retraining on cleared data (same architecture). The repository CODE is Apache-2.0.

Limitations & bias

• Quality targets are partly synthetic (controlled degradations), supplemented by real WIDER negatives; pose labels are 300W-LP's 3DMM-fit GT (synthetic large-pose renders).
• Face models can perform unevenly across demographics, skin tones, and lighting; the training data is not demographically balanced. Evaluate on your own population before use.
• Pose accuracy degrades at extreme angles (|yaw| > 90 deg) and under heavy occlusion.
• fp16 CoreML numerics are verified on-device (iPhone), not on the training box.

Files

• pytorch_model.bin — training checkpoint (state dict).
• model.onnx — opset 17, for servers / cross-platform.
• FaceCaptureQuality.mlpackage — CoreML (fp16), for iOS. Input: RGB image (scale=1/255).
• config.json — input/output spec.

Attribution

• Datasets: 300W-LP and AFLW2000-3D (Zhu et al., "Face Alignment Across Large Poses: A 3D Solution", CVPR 2016; academic / non-commercial), FFHQ (NVIDIA; CC BY-NC-SA 4.0), WIDER FACE (Yang et al., CVPR 2016; CC BY-NC-ND 4.0).
• Verification embeddings for the ERC eval only: InsightFace (buffalo_l).
• Code: https://github.com/shafi-afridi/face-capture-quality
• Demo Space: https://huggingface.co/spaces/shafi-afridi/face-capture-quality-demo