Downloads · 30 days
0
unreal-audio2lipsync
fotonlabs/unreal-audio2lipsync
unreal-audio2lipsync is a audio-to-audio model from fotonlabs. Use it for the audio-to-audio task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as mit.
Audio-driven lipsync model for Unreal Engine MetaHumans. Given a mono 16 kHz audio clip, the model predicts 52-channel ARKit blendshape coefficients at 60 fps — frame-accurate mouth, jaw, tongue, and cheek motion that…
Access
Public
Updated Apr 17, 2026
Repo size
45.8 MB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.pt45.8 MB · 100%
At a glance
- Task
- Audio-to-Audio
- Library
- pytorch
- License
- mit
- Access
- Public
- Created
- Apr 17, 2026
- Updated
- Apr 17, 2026
- SHA
- d8faff2b
Model card
From the Hugging Face model README
Audio2Lipsync
Audio-driven lipsync model for Unreal Engine MetaHumans. Given a mono 16 kHz audio clip, the model predicts 52-channel ARKit blendshape coefficients at 60 fps — frame-accurate mouth, jaw, tongue, and cheek motion that drops straight into the MetaHuman face rig via LiveLink.
Companion code repository: https://github.com/aaryansachdeva/unreal-audio2lipsync
Architecture
- Encoder: HuBERT-Large (frozen) — phonetic features from 16 kHz audio
- Decoder: 8-layer Transformer (d=512, 8 heads, FF=2048, dropout=0.2)
- Output: 52 ARKit blendshape coefficients per frame at 60 fps
- Loss: Weighted masked L1 (mouth channels get 3× weight) + velocity loss
- Training data: iPhone Live Link Face recordings (ARKit blendshapes) paired with their audio tracks
Files
| File | Purpose |
|---|---|
best.pt | EMA inference checkpoint (trainable params only; frozen HuBERT loaded from torchaudio) |
Usage
Install the code repo and point the sidecar server at this checkpoint:
git clone https://github.com/aaryansachdeva/unreal-audio2lipsync.git
cd unreal-audio2lipsync
pip install -r python/requirements.txt
# Download best.pt from this HF repo into a local checkpoints/ dir, then:
python python/src/server.py --ckpt checkpoints/best.pt --stats-dir python/stats
See the code repo README for full Unreal Engine integration instructions.
License
MIT — use commercially, modify, redistribute, no obligations beyond keeping the license notice. See LICENSE in the code repo.
- Task
- Audio-to-Audio
- Library
- pytorch
- License
- mit
- Created
- Apr 17, 2026
- Updated
- Apr 17, 2026