xr-face-mediapipe-pipeline

Face MediaPipe Pipeline

This repository contains a prebuilt SpatialML pipeline package for running MediaPipe Face detection with SecureMR on supported PICO devices.

The package is not only a model file. It bundles the serialized model, SecureML pipeline JSON files, package metadata, and rendering assets needed by an application to construct and run the full pipeline through the utility loader.

Package file

The main artifact is:

xr-face-mediapipe-pipeline.zip

Extracting the zip produces a folder named xr-face-mediapipe-pipeline/ with this structure:

xr-face-mediapipe-pipeline/
├── manifest.json
├── gltf/
│   └── frame.gltf
├── model/
│   └── face_detector.tflite
└── pipeline/
    ├── face_detection_pipeline.json
    └── face_display_pipeline.json

What is included

• manifest.json: package metadata. It declares the pipeline entries, XR mode support, and detection tensor name.
• model/face_detector.tflite: MediaPipe Face detection model in TFLite format (from https://aihub.qualcomm.com/models/mediapipe_face).
• pipeline/face_detection_pipeline.json: SecureMR pipeline definition for camera input, preprocessing, model inference, and JavaScript post-processing.
• pipeline/face_display_pipeline.json: SecureMR pipeline definition for consuming detection output, projecting image-space face coordinates into 3D, and rendering a face-frame overlay.
• gltf/frame.gltf: glTF asset rendered by the display pipeline.

How to use the package

Native SecureMR

Place the extracted xr-face-mediapipe-pipeline/ folder in your app's assets directory. For example, in an Android project:

app/src/main/assets/xr-face-mediapipe-pipeline/

For a native SecureMR integration, your C++ code should refer to the folder by its relative asset path:

constexpr const char* kModelPackageAssetPath = "xr-face-mediapipe-pipeline";

Then load the package with the native SecureMR utility loader:

SecureMR::ModelPackagePipelineBundle bundle;
std::string error;

const bool loaded = SecureMR::SecureMrUtils::LoadModelPackagePipelinesFromAssets(
    kModelPackageAssetPath,
    frameworkSession,
    {},
    bundle,
    error);

In native code, LoadModelPackagePipelinesFromAssets(...) reads manifest.json, loads each referenced pipeline JSON file, patches model operators to use the packaged serialized model, creates shared global tensor bindings between pipelines, and binds packaged glTF assets through pipeline placeholders.

After loading, submit the package pipelines in sequence. This package contains:

1. detection: reads VST camera tensors, runs the MediaPipe Face model, and writes the post_det detection tensor.
2. display: consumes post_det, projects the detection into camera/world space, and renders the frame.gltf face-frame overlay.

Example SecureMR sample

An example application is available in the SecureMR samples repository:

• SecureMR Samples
• Face MediaPipe Pipeline sample

The native sample shows how to download or place this package in app assets and then load it with SecureMrUtils::LoadModelPackagePipelinesFromAssets(...).

In this sample, the display pipeline shows a rectangle and the confidence score over the detected face.

Device support

This package is intended for applications running on supported PICO devices. It requires a runtime environment that supports the bundled pipeline JSON files and serialized model.

License

Apache License 2.0.