samu-xls-r

We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. Unlike previous works on speech representation learning, which learns multilingual contextual speech embedding at the resolution of an acoustic frame (10-20ms), this work focuses on learning multimodal (speech-text) multilingual speech embedding at the resolution of a sentence (5-10s) such that the embedding vector space is semantically aligned across different languages. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR. Although we train SAMU-XLSR with only multilingual transcribed speech data, cross-lingual speech-text and speech-speech associations emerge in its learned representation space. To substantiate our claims, we use SAMU-XLSR speech encoder in combination with a pre-trained LaBSE text sentence encoder for cross-lingual speech-to-text translation retrieval, and SAMU-XLSR alone for cross-lingual speech-to-speech translation retrieval. We highlight these applications by performing several cross-lingual text and speech translation retrieval tasks across several datasets.

Citation

Please cite as:

@ARTICLE{9834099,
  author={Khurana, Sameer and Laurent, Antoine and Glass, James},
  journal={IEEE Journal of Selected Topics in Signal Processing}, 
  title={SAMU-XLSR: Semantically-Aligned Multimodal Utterance-Level Cross-Lingual Speech Representation}, 
  year={2022},
  volume={16},
  number={6},
  pages={1493-1504},
  keywords={Training;Speech processing;Machine translation;Representation learning;Cross-lingual speech representation learning;Language-agnostic speech embedding;zero-shot speech-to-text translation retrieval;zero-shot speech-to-speech translation retrieval},
  doi={10.1109/JSTSP.2022.3192714}}

@inproceedings{khurana2024cross,
  title={Cross-lingual transfer learning for low-resource speech translation},
  author={Khurana, Sameer and Dawalatabad, Nauman and Laurent, Antoine and Vicente, Luis and Gimeno, Pablo and Mingote, Victoria and Glass, James},
  booktitle={2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)},
  pages={670--674},
  year={2024},
  organization={IEEE}
}