Downloads · 30 days
0
shibing624/ai-avatar-wav2lip
ai-avatar-wav2lip is a machine learning model from shibing624. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
<p align="center" <a href="https://github.com/shibing624/AIAvatar" <img src="https://img.shields.io/badge/GitHub-AIAvatar-blue?logo=github" alt="GitHub" </a <a href="https://github.com/lipku/LiveTalking" <img src="htt…
Downloads · 30 days
0
Access
Public
Updated Nov 17, 2025
Repo size
809 MB
Likes
7
Public
Click a slice to open those files.
.zip594 MB · 73%
From the Hugging Face model README
本仓库包含用于 AIAvatar 项目的预训练 Wav2Lip 模型和即用型数字人形象资源。AIAvatar 是基于 LiveTalking 改进的实时交互式流媒体数字人系统,支持高质量的音视频同步对话。
wav2lip.pth (215 MB) - Wav2Lip256 唇形同步生成模型wav2lip_avatar_female_model.zip (353 MB) - 女性数字人形象(含预处理人脸图片)wav2lip_avatar_glass_man.zip (88.4 MB) - 戴眼镜男性数字人形象wav2lip_avatar_long_hair_girl.zip (153 MB) - 长发女性数字人形象AIAvatar 是基于 LiveTalking 改进的数字人解决方案,通过以下特性实现实时唇形同步和自然对话:
✨ 核心功能:
🎥 演示视频:
# 创建 conda 环境
conda create -n aiavatar python=3.10
conda activate aiavatar
# 安装 PyTorch (CUDA 12.4)
conda install pytorch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 pytorch-cuda=12.4 -c pytorch -c nvidia
# 克隆 AIAvatar 仓库
git clone https://github.com/shibing624/AIAvatar.git
cd AIAvatar
# 安装依赖
pip install -r requirements.txt
从本仓库下载文件并配置:
# 创建目录
mkdir -p models data/avatars
# 下载并设置模型
# 将 wav2lip.pth 放到 models/ 目录
cp wav2lip.pth models/
# 下载并解压数字人形象
# 将 wav2lip_avatar_female_model.zip 解压到 data/avatars/
unzip wav2lip_avatar_female_model.zip -d data/avatars/
# 使用 WebRTC 传输启动
sh run.sh
# 在浏览器中打开
# http://<服务器IP>:8010/index.html
注意: 服务器需要开放端口: TCP 8010 和 UDP 1-65536
每个数字人形象包含预处理数据以获得最佳性能:
wav2lip_avatar_female_model/
├── full_imgs/ # 全身图片 (100帧)
│ ├── 0000.png
│ ├── 0001.png
│ └── ...
├── face_imgs/ # 裁剪的人脸区域 (100帧)
│ ├── 0000.png
│ ├── 0001.png
│ └── ...
└── coords.pkl # 人脸边界框坐标
人脸图片规格:
(batch, 1, 80, 16)(batch, 6, H, W) (遮罩 + 原图)| GPU 型号 | FPS (Batch=8) | 延迟 |
|---|---|---|
| RTX 3060 | 60 | ~13ms |
| RTX 3080 Ti | 120 | ~6.5ms |
| RTX 4090 | 150+ | ~5ms |
测试环境: Ubuntu 24.04, PyTorch 2.5.0, CUDA 12.4
from src.lipreal import LipReal
# 初始化数字人
digital_human = LipReal(
model_path="models/wav2lip.pth",
avatar_id="wav2lip_avatar_female_model",
batch_size=8
)
# 从音频生成唇形同步视频
digital_human.infer(audio_path="input.wav")
在专用 GPU 服务器上运行 Wav2Lip:
# GPU 服务器上
python src/gpu_wav2lip_service.py --port 8888
# 客户端机器上 (仅需CPU)
python main.py --gpu_server http://gpu-server:8888
将 CPU 客户端和 GPU 服务器分离以更好地利用资源:
┌─────────────┐ HTTP/HTTPS ┌──────────────┐
│ CPU 客户端 │ ────────────────────────> │ GPU 服务器 │
│ (WebRTC) │ <──────────────────────── │ (Wav2Lip) │
└─────────────┘ Mel + 帧数据 └──────────────┘
支持多个用户同时使用,并优化内存使用:
如果你在研究或项目中使用这些模型,请引用:
@software{aiavatar2024,
author = {Xu Ming},
title = {AIAvatar: Improved Real-time Interactive Streaming Digital Human},
year = {2024},
url = {https://github.com/shibing624/AIAvatar}
}
@software{livetalking2024,
author = {lipku},
title = {LiveTalking: Real-time Interactive Streaming Digital Human},
year = {2024},
url = {https://github.com/lipku/LiveTalking}
}
@inproceedings{wav2lip2020,
title={A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild},
author={Prajwal, KR and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P and Jawahar, CV},
booktitle={ACM Multimedia},
year={2020}
}