hunyuanvideo-foley-models

<div align="center"> <img src="assets/logo.png" alt="HunyuanVideo-Foley Logo" width="400"> <h4>Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation</h4> <p align="center"> <strong>Professional-grade AI sound effect generation for video content creators</strong> </p> <div align="center" style="margin: 20px 0;"> <a href=https://github.com/Tencent-Hunyuan/HunyuanVideo-Foley target="_blank"><img src=https://img.shields.io/badge/Code-black.svg?logo=github height=22px></a> <a href=https://szczesnys.github.io/hunyuanvideo-foley target="_blank"><img src=https://img.shields.io/badge/Page-bb8a2e.svg?logo=github height=22px></a> <a href=https://huggingface.co/tencent/HunyuanVideo-Foley target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Models-d96902.svg height=22px></a> <a href=https://huggingface.co/spaces/tencent/HunyuanVideo-Foley target="_blank"><img src=https://img.shields.io/badge/%F0%9F%A4%97%20Demo-276cb4.svg height=22px></a> <a href=https://arxiv.org/abs/2508.16930 target="_blank"><img src=https://img.shields.io/badge/Report-b5212f.svg?logo=arxiv height=22px></a> <a href=https://x.com/TencentHunyuan target="_blank"><img src=https://img.shields.io/badge/Hunyuan-black.svg?logo=x height=22px></a> </div> </div>

---

<div align="center">

👥 Authors

<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); padding: 20px; border-radius: 15px; margin: 20px 0;">

Sizhe Shan^1,2* • Qiulin Li^1,3* • Yutao Cui¹ • Miles Yang¹ • Yuehai Wang² • Qun Yang³ • Jin Zhou^1† • Zhao Zhong¹

</div> <div style="margin-top: 15px; font-size: 14px; color: #666;">

🏢 ¹Tencent Hunyuan • 🎓 ²Zhejiang University • ✈️ ³Nanjing University of Aeronautics and Astronautics

*Equal contribution • †Project lead

</div> </div>

---

🔥🔥🔥 News

<div style="padding: 20px; border-radius: 15px; margin: 20px 0; border-left: 5px solid;">

• [2025.9.29] 🚀 HunyuanVideo-Foley-XL Model Release - Release XL-sized model with offload inference support, significantly reducing VRAM requirements.
• [2025.8.28] 🌟 HunyuanVideo-Foley Open Source Release - Inference code and model weights publicly available.

</div>

---

<!-- ## 🎥 **Demo & Showcase** --> <!-- --> <!-- <div align="center"> --> <!-- --> <!-- > **Experience the magic of AI-generated Foley audio in perfect sync with video content!** --> <!-- --> <!-- <div style="border: 3px solid #4A90E2; border-radius: 15px; padding: 10px; margin: 20px 0; background: linear-gradient(135deg, #f5f7fa 0%, #c3cfe2 100%);"> --> <!-- --> <!-- <video src="https://github.com/user-attachments/assets/087a4b59-8b22-4b7a-bac3-8f5d5a9972fe" width="80%" controls style="border-radius: 10px; box-shadow: 0 8px 32px rgba(0,0,0,0.1);"> </video> --> <!-- --> <!-- <p><em>🎬 Watch how HunyuanVideo-Foley generates immersive sound effects synchronized with video content</em></p> --> <!-- --> <!-- </div> -->

✨ Key Highlights

<table align="center" style="border: none; margin: 20px 0;"> <tr> <td align="center" width="33%">

🎭 Multi-scenario Sync
High-quality audio synchronized with complex video scenes

</td> <td align="center" width="33%">

🧠 Multi-modal Balance
Perfect harmony between visual and textual information

</td> <td align="center" width="33%">

🎵 48kHz Hi-Fi Output
Professional-grade audio generation with crystal clarity

</td> </tr> </table> </div>

---

📄 Abstract

<div align="center" style="background: linear-gradient(135deg, #ffeef8 0%, #f0f8ff 100%); padding: 30px; border-radius: 20px; margin: 20px 0; border-left: 5px solid #ff6b9d; color: #333;">

🚀 Tencent Hunyuan open-sources HunyuanVideo-Foley an end-to-end video sound effect generation model!

A professional-grade AI tool specifically designed for video content creators, widely applicable to diverse scenarios including short video creation, film production, advertising creativity, and game development.

</div>

🎯 Core Highlights

<div style="display: grid; grid-template-columns: 1fr; gap: 15px; margin: 20px 0;"> <div style="border-left: 4px solid #4CAF50; padding: 15px; background: #f8f9fa; border-radius: 8px; color: #333;">

🎬 Multi-scenario Audio-Visual Synchronization
Supports generating high-quality audio that is synchronized and semantically aligned with complex video scenes, enhancing realism and immersive experience for film/TV and gaming applications.

</div> <div style="border-left: 4px solid #2196F3; padding: 15px; background: #f8f9fa; border-radius: 8px; color: #333;">

⚖️ Multi-modal Semantic Balance
Intelligently balances visual and textual information analysis, comprehensively orchestrates sound effect elements, avoids one-sided generation, and meets personalized dubbing requirements.

</div> <div style="border-left: 4px solid #FF9800; padding: 15px; background: #f8f9fa; border-radius: 8px; color: #333;">

🎵 High-fidelity Audio Output
Self-developed 48kHz audio VAE perfectly reconstructs sound effects, music, and vocals, achieving professional-grade audio generation quality.

</div> </div> <div align="center" style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 20px; border-radius: 15px; margin: 20px 0; color: #333;">

🏆 SOTA Performance Achieved

HunyuanVideo-Foley comprehensively leads the field across multiple evaluation benchmarks, achieving new state-of-the-art levels in audio fidelity, visual-semantic alignment, temporal alignment, and distribution matching - surpassing all open-source solutions!

</div> <div align="center">

📊 Performance comparison across different evaluation metrics - HunyuanVideo-Foley leads in all categories

</div>

---

🔧 Technical Architecture

📊 Data Pipeline Design

<div align="center" style="margin: 20px 0;">

🔄 Comprehensive data processing pipeline for high-quality text-video-audio datasets

</div> <div style="background: #f8f9fa; padding: 20px; border-radius: 10px; border-left: 4px solid #17a2b8; margin: 20px 0; color: #333;">

The TV2A (Text-Video-to-Audio) task presents a complex multimodal generation challenge requiring large-scale, high-quality datasets. Our comprehensive data pipeline systematically identifies and excludes unsuitable content to produce robust and generalizable audio generation capabilities.

</div>

🏗️ Model Architecture

<div align="center" style="margin: 20px 0;">

🧠 HunyuanVideo-Foley hybrid architecture with multimodal and unimodal transformer blocks

</div> <div style="background: #f8f9fa; padding: 20px; border-radius: 10px; border-left: 4px solid #28a745; margin: 20px 0; color: #333;">

HunyuanVideo-Foley employs a sophisticated hybrid architecture:

• 🔄 Multimodal Transformer Blocks: Process visual-audio streams simultaneously
• 🎵 Unimodal Transformer Blocks: Focus on audio stream refinement
• 👁️ Visual Encoding: Pre-trained encoder extracts visual features from video frames
• 📝 Text Processing: Semantic features extracted via pre-trained text encoder
• 🎧 Audio Encoding: Latent representations with Gaussian noise perturbation
• ⏰ Temporal Alignment: Synchformer-based frame-level synchronization with gated modulation

</div>

---

📈 Performance Benchmarks

🎬 MovieGen-Audio-Bench Results

<div align="center">

Objective and Subjective evaluation results demonstrating superior performance across all metrics

</div> <div style="overflow-x: auto; margin: 20px 0;"> </div>

🎯 Kling-Audio-Eval Results

<div align="center">

Comprehensive objective evaluation showcasing state-of-the-art performance

</div> <div style="overflow-x: auto; margin: 20px 0;"> </div> <div align="center" style="background: linear-gradient(135deg, #4CAF50 0%, #45a049 100%); color: white; padding: 15px; border-radius: 10px; margin: 20px 0;">

🎉 Outstanding Results! HunyuanVideo-Foley achieves the best scores across ALL evaluation metrics, demonstrating significant improvements in audio quality, synchronization, and semantic alignment.

</div>

---

🚀 Quick Start

📦 Installation

<div style="background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 20px; border-radius: 15px; margin: 20px 0;">

🔧 System Requirements

• CUDA: 12.4 or 11.8 recommended
• Python: 3.8+
• OS: Linux (primary support)

</div>

Step 1: Clone Repository

# 📥 Clone the repository
git clone https://github.com/Tencent-Hunyuan/HunyuanVideo-Foley
cd HunyuanVideo-Foley

Step 2: Environment Setup

<div style="background: #fff3cd; padding: 15px; border-radius: 8px; border-left: 4px solid #ffc107; margin: 10px 0; color: #333;">

💡 Tip: We recommend using Conda for Python environment management.

</div>

# 🔧 Install dependencies
pip install -r requirements.txt

Step 3: Download Pretrained Models

<div style="background: #d1ecf1; padding: 15px; border-radius: 8px; border-left: 4px solid #17a2b8; margin: 10px 0; color: #333;">

🔗 Download Model weights from Huggingface

# using git-lfs
git clone https://huggingface.co/tencent/HunyuanVideo-Foley

# using huggingface-cli
huggingface-cli download tencent/HunyuanVideo-Foley

</div>

---

💻 Usage

🎬 Single Video Generation

<div style="background: #e8f5e8; padding: 15px; border-radius: 8px; border-left: 4px solid #28a745; margin: 10px 0; color: #333;">

Generate Foley audio for a single video file with text description:

</div>

python3 infer.py \
    --model_path PRETRAINED_MODEL_PATH_DIR \
    --config_path ./configs/hunyuanvideo-foley-xxl.yaml \
    --single_video video_path \
    --single_prompt "audio description" \
    --output_dir OUTPUT_DIR

📂 Batch Processing

<div style="background: #fff3e0; padding: 15px; border-radius: 8px; border-left: 4px solid #ff9800; margin: 10px 0; color: #333;">

Process multiple videos using a CSV file with video paths and descriptions:

</div>

python3 infer.py \
    --model_path PRETRAINED_MODEL_PATH_DIR \
    --config_path ./configs/hunyuanvideo-foley-xxl.yaml \
    --csv_path assets/test.csv \
    --output_dir OUTPUT_DIR

🌐 Interactive Web Interface

<div style="background: #f3e5f5; padding: 15px; border-radius: 8px; border-left: 4px solid #9c27b0; margin: 10px 0; color: #333;">

Launch a user-friendly Gradio web interface for easy interaction:

</div>

export HIFI_FOLEY_MODEL_PATH=PRETRAINED_MODEL_PATH_DIR
python3 gradio_app.py

<div align="center" style="margin: 20px 0; color: #333;">

🚀 Then open your browser and navigate to the provided local URL to start generating Foley audio!

</div>

---

📚 Citation

<div style="background: #f8f9fa; padding: 20px; border-radius: 10px; border-left: 4px solid #6c757d; margin: 20px 0; color: #333;">

If you find HunyuanVideo-Foley useful for your research, please consider citing our paper:

</div>

@misc{shan2025hunyuanvideofoleymultimodaldiffusionrepresentation,
      title={HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation}, 
      author={Sizhe Shan and Qiulin Li and Yutao Cui and Miles Yang and Yuehai Wang and Qun Yang and Jin Zhou and Zhao Zhong},
      year={2025},
      eprint={2508.16930},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2508.16930}, 
}

---

🙏 Acknowledgements

<div align="center">

We extend our heartfelt gratitude to the open-source community!

</div> <table align="center" style="width: 100%; border: none; margin: 20px 0;"> <tr> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

🎨 Stable Diffusion 3
Foundation diffusion models

</td> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

⚡ FLUX
Advanced generation techniques

</td> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

🎵 MMAudio
Multimodal audio generation

</td> </tr> <tr> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

🤗 HuggingFace
Platform & diffusers library

</td> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

🗜️ DAC
High-Fidelity Audio Compression

</td> <td align="center" style="width: 33%; padding: 10px; vertical-align: top;">

🔗 Synchformer
Audio-Visual Synchronization

</td> </tr> </table> <div align="center" style="background: linear-gradient(135deg, #74b9ff 0%, #0984e3 100%); color: white; padding: 20px; border-radius: 15px; margin: 20px 0;">

🌟 Special thanks to all researchers and developers who contribute to the advancement of AI-generated audio and multimodal learning!

</div>

---

<div align="center" style="margin: 30px 0;">

🔗 Connect with Us

<p style="color: #666; margin-top: 15px; font-size: 14px;">

© 2025 Tencent Hunyuan. All rights reserved. | Made with ❤️ for the AI community

</p> </div>