WAVE-7B

WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

<!-- <div align="center"> --> <div> <a href="https://arxiv.org/pdf/2509.21990" target="_blank"> <img src="https://img.shields.io/badge/Paper-arXiv-red.svg" alt="Paper arXiv"> </a> <a href="https://huggingface.co/tsinghua-ee/WAVE-7B" target="_blank"> <img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-yellow" alt="Hugging Face Models"> </a> <a href="https://github.com/TCL606/WAVE" target="_blank"> <img src="https://img.shields.io/badge/GitHub-Code-blue" alt="GitHub Code"> </a> <img src="https://img.shields.io/badge/License-Apache%202.0-green" alt="License"> </div> <!-- </div> -->

---

🔥 News

• [2026/02/11]: 🚀 We released the official code and model checkpoints!
• [2026/02/06]: 🎉 WAVE has been accepted to ICLR 2026 as an Oral Presentation!

---

📖 Abstract

We introduce WAVE (Unified & Versatile Audio-Visual Embeddings), the first LLM-based embedding model that creates a unified representation space for text, audio, silent video, and synchronized audio-visual modalities.

Unlike previous methods that use separate encoders, WAVE is built upon Qwen2.5-Omni and employs a novel hierarchical feature fusion strategy. It features a dual-encoder design for audio (Speech + Environmental Sounds) and is trained via a joint multi-modal, multi-task approach.

Key Capabilities:

1. Any-to-Any Retrieval: Seamlessly retrieve across text, audio, and video modalities.
2. Prompt-Aware Embeddings: Generate embeddings tailored to user instructions (e.g., specific questions), significantly boosting performance in multimodal QA.
3. SOTA Performance: WAVE achieves state-of-the-art results on the MMEB-v2-video benchmark and excels in audio/audio-visual retrieval tasks.

<div align="center"> <img src="https://github.com/TCL606/WAVE/blob/main/assets/wave.jpg?raw=true" width="90%" alt="WAVE Architecture"/> <br> <em>Overview of WAVE Architecture. It supports text-only, visual-only, audio-only, and audio-visual inputs.</em> </div>

---

🏆 Performance

WAVE achieves SOTA performance on MMEB-v2-video, outperforming industrial-grade models like Seed-1.6-Embedding.

WAVE also demonstrates strong performance across audio and audio-visual modalities.

🛠️ Installation

For key library versions, please refer to requirements.txt. Install via:

pip install -r requirements.txt

---

🤖 Model Zoo

We provide the model checkpoint on Hugging Face.

---

🚀 Quick Start

To evaluate WAVE on your own datasets or benchmarks, follow these steps:

1. Data Preparation: Organize your data into the sample JSON format. Refer to the templates provided in the scripts/ directory, such as:

   • scripts/ret_av2t.json (video/audio to text retrieval)
   • scripts/ret_mc.json (QA)
   • scripts/ret_v2a.json (audio-visual retrieval)

2. Inference: Modify scripts/direct_test.sh, then run the script:

   bash scripts/direct_test.sh
   

   The results will be automatically saved in the output/test directory.

---

📝 Citation

If you find WAVE useful for your research, please consider citing our paper:

@inproceedings{tang2026wave,
    title={{WAVE}: Learning Unified \& Versatile Audio-Visual Embeddings with Multimodal {LLM}},
    author={Changli Tang and Qinfan Xiao and Ke Mei and Tianyi Wang and Fengyun Rao and Chao Zhang},
    booktitle={The Fourteenth International Conference on Learning Representations},
    year={2026},
    url={https://openreview.net/forum?id=MiV3WXDYJb}
}

---

License: Apache 2.0