Llama-speechlmm-1.0-s

Model information

The SpeechLMM 1.0 collection of multimodal and multilingual large language models is a collection of instruction-tuned generative models in 4 different sizes: S (2B), M (4B), L (9B) and XL (71B), supporting text, audio and video as input and only text as output. The SpeechLMM 1.0 models are optimized for various X-to-text generation tasks, namely:

• Machine Translation
• Automatic Speech Recognition
• Speech Translation
• Speech Summarization
• Spoken Question Answering
• Spoken Language Understanding (beta)
• Visual Speech Recognition (beta)

Model Developer: Meetween consortium

Supported Languages: English, French, Italian, German, and Spanish are officially supported (for a subset of the supported tasks). The Llama 3.X backbone and the SeamlessM4T v2 audio encoder have been trained on a broader collection of languages than these 5 supported languages, so the model might exhibit good performance on other languages too.

Model Release Date: Feb 28, 2025

License: see LICENSE

Model Architecture

SpeechLMM 1.0 an auto-regressive multimodal language model based on a Llama 3.X backbone (X varies with the model size), a speech-specific stack consisting of a pre-trained audio encoder (SeamlessM4T v2) and an audio adapter, and a video-specific stack consisting of a pre-trained video encoder (Auto-AVSR) and a video adapter.

<!-- TODO: add the image of the model architecture here -->

Audio and video encoders

For all the 4 sizes of SpeechLMM 1.0, the audio encoder is SeamlessM4T v2 Large (facebook/seamless-m4t-v2-large) and the video encoder is Auto-AVSR (vsr_trlrs3vox2_base).

Audio and video adapters

For all the 4 sizes of SpeechLMM 1.0, the audio and video adapters are:

LLM backbone

Training Details

For a comprehensive training journal including detailed metrics, hyperparameters, and training curves for all SpeechLMM 1.0 variants, see our W&B Training Report.

How to use

Currently, this model can only be used via our speechlmm codebase. Refer to the instructions there for more details.

Important: before you can use this model, you must download the SeamlessM4T v2 speech encoder and the Auto-AVSR video encoder by following the instructions provided in the README of the above repo. Please note that by doing so, you agree with their respective license terms.

Training Data

Monolingual

Multilingual

Evaluation Results

The following results specifically refer to the S model.

ASR Metrics

SQA Metrics

NOTE: Accuracy is measured with an LLM as a judge (Llama3-70b-8192, via the Groq API) using the following prompts:

• System prompt

  You are a helpful assistant that evaluates answers to questions given a certain context. You will be given inputs of the form:<br> Context: <CONTEXT><br> Question: <QUESTION><br> Answer: <ANSWER><br> Your task is to determine if the given answer is correct or not, assuming the correct answer is contained in the context. Your response should be formatted as a JSON string having the following structure: {"correct_answer": <true/false>, "rationale": <RATIONALE>} where 'rationale' must be a string explaining why the answer is correct or incorrect. If you need to include double quote characters (") in the 'rationale' string, you must escape them with a backslash (\). For example, if you want to include the string "Hello, World!", you should write it as \"Hello, World!\".

• User prompt

  Context: <CONTEXT><br> Question: <QUESTION><br> Answer: <ANSWER>

MT Metrics

SSUM Metrics

ST Metrics

Framework versions

• Transformers 4.45.0
• Pytorch 2.3.1+cu124.post2
• Datasets 3.2.0
• Tokenizers 0.20.0