Audio2Face-3D-v2.3.1-James-MLX

Audio2Face-3D v2.3.1 James — MLX

NVIDIA Audio2Face-3D v2.3.1 (James) exported for the MLX Swift runtime in speech-swift. Speech audio in, timestamped facial-animation coefficients out — fully on-device on Apple Silicon.

The bundle contains the learned tensors from the official ONNX checkpoint in safetensors form plus the NVIDIA runtime metadata. The hand-written MLX graph in speech-swift runs the real model forward pass; outputs are parity-checked against the official ONNX model.

Output

Each frame carries 169 coefficients: 140 skin blendshapes, 10 tongue blendshapes, 15 jaw controls, and 4 eye controls, with a 10-value explicit + 16-value implicit emotion input (16 kHz mono audio, 8320-sample windows).

Files

Usage (Swift)

import Audio2Face3D

let model = try await Audio2Face3DModel.fromPretrained(
    modelId: "aufklarer/Audio2Face-3D-v2.3.1-James-MLX")
let frames = try model.frames(for: samples, sampleRate: 16_000)

Or from the speech CLI:

speech avatar-motion input.wav --output frames.jsonl

Identities

• Audio2Face-3D-v2.3-Mark-MLX — 301 coefficients (272 skin)
• Audio2Face-3D-v2.3.1-Claire-MLX — 169 coefficients (140 skin)
• Audio2Face-3D-v2.3.1-James-MLX (this repo, runtime default) — 169 coefficients (140 skin)

Coefficients are identity-specific: renderers need the matching rig or a retarget projection for the chosen identity.

License and attribution

The weights are NVIDIA's Audio2Face-3D v2.3.1 James model, redistributed in MLX-ready form under the NVIDIA Open Model License. All model credit goes to NVIDIA.

Ecosystem

• soniqo.audio — documentation and use cases
• speech-swift — MLX Swift runtime (this model's target)
• speech-core — C++ engine
• speech-android — Android engine

Other MLX models in this collection

See the MLX Speech Models collection.