Downloads · 30 days
976
39% of all-time downloads
MuseTalk-1.5-fp16
mlx-community/MuseTalk-1.5-fp16
MuseTalk-1.5-fp16 is a image-to-image model from mlx-community. Use it when you need one image transformed into another. It is set up for mlx. The card lists the license as mit.
Apple-MLX port of MuseTalk 1.5 (TMElyralab / Tencent Music) — realtime, high-quality lip-sync via single-step latent-space inpainting (not diffusion). Runs natively on Apple Silicon. MIT-licensed, commercial use OK.
All-time downloads
2.5K
Public
Repo size
1.9 GB
Likes
6
Public
Repo makeup
Click a slice to open those files.
.safetensors1.9 GB · 100%
At a glance
- Task
- Image-to-Image
- Library
- mlx
- License
- mit
- Access
- Public
- Created
- Jun 5, 2026
- Updated
- Jun 5, 2026
- SHA
- ad54104a
Base models
Model card
From the Hugging Face model README
MuseTalk 1.5 — MLX (fp16)
Apple-MLX port of MuseTalk 1.5 (TMElyralab / Tencent Music) — realtime, high-quality lip-sync via single-step latent-space inpainting (not diffusion). Runs natively on Apple Silicon. MIT-licensed, commercial use OK.
This variant: Full fp16 (VAE + UNet + Whisper encoder). Decoded-face error vs the PyTorch reference: mean |Δ| ≈ 0.32/255.
Components (all in this repo, self-contained, torch-free)
| File | What |
|---|---|
unet.safetensors | SD1.x UNet2DConditionModel (in=8, out=4, cross_attn=384), single-step t=0 |
vae.safetensors | sd-vae-ft-mse AutoencoderKL (fp16) |
whisper_encoder.safetensors | whisper-tiny audio encoder (fp16) |
config.json | dtype / quantization / scaling factor |
Performance
Realtime on an M-series GPU: ~34 generated 256² faces/sec at batch 8 (>25 fps video rate), ~7 GB peak. fp16 inference.
Usage
from musetalk_mlx.pipeline_mlx import MuseTalkPipeline
pipe = MuseTalkPipeline.from_pretrained_mlx("MuseTalk-1.5-MLX-fp16")
# crop_bgr: a 256x256 face crop; chunks: (N,50,384) whisper audio features
latents = pipe.get_latents_for_unet(crop_bgr)
faces = pipe.generate_faces(latents, audio_chunks) # BGR uint8 lip-synced faces
Face detection / cropping / paste-back blending use the upstream (MuseTalk) CPU preprocessing.
Parity (vs PyTorch, cpu fp32)
VAE encode 1.7e-5 · decode 3.4e-5 · UNet forward 1.4e-6 · whisper encoder 1.6e-5 · face-level e2e recon ≤ 2/255.
License
MIT (mirrors upstream MuseTalk). Dependency models keep their own permissive licenses. Port by MVS Collective (xocialize-code).
- Task
- Image-to-Image
- Library
- mlx
- License
- mit
- Created
- Jun 5, 2026
- Updated
- Jun 5, 2026