fama-medium

FAMA-medium

<div> <img src="FAMA.png" width="100%" alt="FAMA" /> </div>

Table of Contents

1. Overview
2. Usage
3. Results
4. License
5. Citation

Overview

FAMA is the first family of large-scale open-science SFMs for English and Italian trained on over 150k hours of exclusively open-source(OS)-compliant speech data.

FAMA models achieve remarkable results, with ASR and ST improvements on average across languages compared to OWSM, and is competitive in terms of ASR performance with the Whisper model family while being up to 8 times faster.

All the artifacts used for realizing FAMA models, including codebase, datasets, and models themself are released under OS-compliant licenses, promoting a more responsible creation of models in our community.

It is available in 2 sizes, with 2 variants for ASR only:

• FAMA-small - 475 million parameters
• FAMA-medium - 878 million parameters
• FAMA-small-asr - 475 million parameters
• FAMA-medium-asr - 878 million parameters

For further details, please refer to the paper FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian. The code is available in the Github repository.

Usage

FAMA models are supported in Hugging Face 🤗 Transformers. To run the model, first install the Transformers and Datasets libraries.

pip install transformers==4.48.1 datasets

To perform a single inference on a sample audio file using the pipeline class, run:

import torch
from transformers import AutoProcessor, pipeline
from datasets import load_dataset

model_id = "FBK-MT/fama-medium"
processor = AutoProcessor.from_pretrained(model_id)

device = "cuda:0" if torch.cuda.is_available() else "cpu"
tgt_lang = "en"

# Force the model to start with the language tag
lang_tag = "<lang:{}>".format(tgt_lang)
lang_tag_id = processor.tokenizer.convert_tokens_to_ids(lang_tag)

generate_kwargs = {"num_beams": 5, "no_repeat_ngram_size": 5, "forced_bos_token_id": lang_tag_id}

pipe = pipeline(
    "automatic-speech-recognition",
    model=model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device=device,
    return_timestamps=False,
    generate_kwargs=generate_kwargs
)

dataset = load_dataset("distil-whisper/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[0]["audio"]

result = pipe(sample)
print(result["text"])

Where tgt_lang is the target language (either en or it). The source languages has not to be specified. To run the inference on a local audio file audio.wav, call the pipeline with:

result = pipe("audio.wav")

To perform a batch inference with size batch_size, run:

result = pipe(["audio_1.wav", "audio_2.wav"], batch_size=2)

For the inference, we suggest converting the audio files in wav format with 16kHz sampling rate and 1 channel.

Results

We evaluate FAMA on ASR and ST tasks using popular open-source datasets such as CommonVoice, Multilingual LibriSpeech (MLS), VoxPopuli, CoVoST2 and FLEURS. The metrics used are WER (↓) for ASR, and COMET (↑) for ST.

We also benchmark FAMA in terms of computational time and maximum batch size supported on HuggingFace against Whisper and SeamlessM4T models. The metric used is the inverse real time factor (xRTF).

Key highlights:

• FAMA achieves up to 4.2 WER and 0.152 COMET improvement on average across languages compared to OWSM v3.1
• FAMA is up to 8 times faster than Whisper large-v3 while achieving comparable ASR performance

Automatic Speech Recogniton (ASR)

Speech Translation (ST)

Computational Time and Maximum Batch Size

License

We release the FAMA model weights, and training data under the CC-BY 4.0 license. The training data can be found in FAMA Training Data. The original FBK-fairseq codebase used to train the model is released under the Apache 2.0 license.

Citation

If you use FAMA in your work, please cite:

@misc{papi2025fama,
      title={FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian}, 
      author={Sara Papi and Marco Gaido and Luisa Bentivogli and Alessio Brutti and Mauro Cettolo and Roberto Gretter and Marco Matassoni and Mohamed Nabih and Matteo Negri},
      year={2025}
}