Llama-speechlmm-1.0-l

Model information

The SpeechLMM 1.0 collection of multimodal and multilingual large language models is a collection of instruction-tuned generative models in 4 different sizes: S (2B), M (4B), L (9B) and XL (71B), supporting text, audio and video as input and only text as output. The SpeechLMM 1.0 models are optimized for various X-to-text generation tasks, namely:

• Machine Translation
• Automatic Speech Recognition
• Speech Translation
• Speech Summarization
• Spoken Question Answering
• Spoken Language Understanding
• Visual Speech Recognition
• Text Summarization

Model Developer: Meetween consortium

Supported Languages: English, French, Italian, German, and Spanish are officially supported (for a subset of the supported tasks). The Llama 3.X backbone and the SeamlessM4T v2 audio encoder have been trained on a broader collection of languages than these 5 supported languages, so the model might exhibit good performance on other languages too.

Model Release Date: Feb 28, 2025

License: see LICENSE

Model Architecture

SpeechLMM 1.0 an auto-regressive multimodal language model based on a Llama 3.X backbone (X varies with the model size), a speech-specific stack consisting of a pre-trained audio encoder (SeamlessM4T v2) and an audio adapter, and a video-specific stack consisting of a pre-trained video encoder (Auto-AVSR) and a video adapter.

<!-- TODO: add the image of the model architecture here -->

Audio and video encoders

For all the 4 sizes of SpeechLMM 1.0, the audio encoder is SeamlessM4T v2 Large (facebook/seamless-m4t-v2-large) and the video encoder is Auto-AVSR (vsr_trlrs3vox2_base).

Audio and video adapters

For all the 4 sizes of SpeechLMM 1.0, the audio and video adapters are:

LLM backbone

Training Details

For a comprehensive training journal including detailed metrics, hyperparameters, and training curves for all SpeechLMM 1.0 variants, see our W&B Training Report.

How to use

Currently, this model can only be used via our speechlmm codebase. Refer to the instructions there for more details.

Important: before you can use this model, you must download the SeamlessM4T v2 speech encoder and the Auto-AVSR video encoder by following the instructions provided in the README of the above repo. Please note that by doing so, you agree with their respective license terms.

Training Data

Monolingual

Multilingual

Evaluation Results

The following results specifically refer to the L model.

ASR Metrics

SLU Metrics

SQA Metrics

SSUM Metrics

ST Metrics

LIPREAD Metrics

MT Metrics

TSUM Metrics

Framework versions

• Transformers 4.45.0
• Pytorch 2.3.1+cu124.post2
• Datasets 3.2.0
• Tokenizers 0.20.0