M2SE-VTTS

M²SE-VTTS: Multi-Modal and Multi-Scale Spatial Environment Understanding for Immersive Visual Text-to-Speech

Visual Text-to-Speech (VTTS) aims to take the environmental image as the prompt to synthesize the reverberant speech for the spoken content. The challenge of this task lies in understanding the spatial environment from the image. Many attempts have been made to extract global spatial visual information from the RGB space of a spatial image. However, local and depth image information are crucial for understanding the spatial environment, which previous works have ignored. To address the issues, we propose a novel multi-modal and multi-scale spatial environment understanding scheme to achieve immersive VTTS, termed M²SE-VTTS. The multi-modal aims to take both the RGB and Depth spaces of the spatial image to learn more comprehensive spatial information, and the multi-scale seeks to model the local and global spatial knowledge simultaneously. Specifically, we first split the RGB and Depth images into patches and adopt the Gemini-generated environment captions to guide the local spatial understanding. After that, the multi-modal and multi-scale features are integrated by the local-aware global spatial understanding. In this way, M²SE-VTTS effectively models the interactions between local and global spatial contexts in the multi-modal spatial environment. Objective and subjective evaluations suggest that our model outperforms the advanced baselines in environmental speech generation.

<p align="center"> <img src="assets/architecture.png" width="100%" alt="M2SE-VTTS Architecture"> </p>

Repository Contents

Usage

Please refer to the GitHub repository for installation, training, and inference instructions.

git clone https://github.com/he-shuwei/M2SE-VTTS.git
cd M2SE-VTTS

# Download checkpoints
# Place m2se_vtts/, pretrain_encoder/, pretrain_decoder/, and bigvgan/ under checkpoints/
# Place captions under data/raw_data/captions/
# Extract mfa_outputs.tar.gz to data/processed_data/mfa/outputs/

# Inference
bash scripts/infer/run_infer.sh \
    --ckpt checkpoints/m2se_vtts/model_ckpt_best.pt \
    --outdir results/m2se_vtts/test_seen \
    --batch_size 16

Citation

@inproceedings{liu2025multi,
  title={Multi-modal and multi-scale spatial environment understanding for immersive visual text-to-speech},
  author={Liu, Rui and He, Shuwei and Hu, Yifan and Li, Haizhou},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={23},
  pages={24632--24640},
  year={2025}
}

Acknowledgments

This project builds upon several excellent open-source projects:

• NATSpeech — non-autoregressive TTS framework
• DiffSinger — diffusion-based acoustic model
• F5-TTS — Diffusion Transformer (DiT) architecture
• BigVGAN — neural vocoder by NVIDIA
• SoundSpaces-Speech — audio-visual dataset by Meta Research
• CLIP — visual-language encoder by OpenAI
• RMVPE — robust pitch extractor