musetalk-models

MuseTalk V15 UNet — AEmotionStudio Mirror

Mirror of the MuseTalk V15 UNet weights for use with ComfyUI-FFMPEGA.

About

MuseTalk is a real-time, high-quality lip sync model that synchronizes lip movements in video to match provided audio. It supports:

• Video + Audio lip sync — make a person in a video speak new dialogue
• Image + Audio talking head — animate a portrait photo with speech audio
• Multi-face support — sync multiple faces in a single video
• Batch inference — process multiple frames simultaneously for speed

Files

Usage with ComfyUI-FFMPEGA

This model is auto-downloaded when you use the lip_sync skill in ComfyUI-FFMPEGA.

Example Prompts

Lip sync this video to the provided audio

Make the person's lips match the speech

Dub this video with the new voiceover

The fp16 variant is preferred by default when use_float16 is enabled (default). Falls back to fp32 if fp16 is unavailable.

Manual Download

If auto-download is disabled, download the files and place them in:

ComfyUI/models/musetalk/musetalkV15/

Additional Dependencies

MuseTalk also requires these models (auto-downloaded from HuggingFace on first use):

• SD-VAE (stabilityai/sd-vae-ft-mse) — ~335 MB
• Whisper-tiny (openai/whisper-tiny) — ~75 MB

VRAM Requirements

• Minimum: ~4 GB
• Recommended: ~6 GB
• Uses subprocess isolation to prevent CUDA memory leaks

License

• MuseTalk code: MIT License
• SD-VAE: CreativeML Open RAIL-M
• Whisper: MIT License

Citation

@article{zhang2024musetalk,
  title={MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting},
  author={Zhang, Yue and Liu, Minhao and Chen, Zhaokang and Wu, Bin and others},
  journal={arXiv preprint arXiv:2410.10122},
  year={2024}
}