whisper-large-v3-pt

Model Card for Finetuned Version of Whisper-Large-V3

This model was trained on a subset of the synthetically generated data that later on was filtered to increase the performance of Whisper Model. The approach involves aligning representations of synthetic audio and corresponding text transcripts to identify and remove low-quality samples, improving the overall training data quality.

In this Specific Model we used 96,08% of synthetic data generated by SeamllesMT4LargeV2, the rest was removed by the filtering model. The training set also contained, the CommonVoice Dataset, Multilibri Speach, and Bracarense (Fully Portuguese Dialect)

Model Details

• Developed by: Yuriy Perezhohin, Tiago Santos, Victor Costa, Fernando Peres, and Mauro Castelli.
• Funded by: Remynd
• Shared by: Remynd
• Model type: ASR with contrastive learning-based synthetic data filtering
• Language: Portuguese
• License: APACHE 2.0
• Finetuned from model: Whisper Large V3

Model Sources

• Repository: https://github.com/my-north-ai/semantic_audio_filtering
• Paper: https://ieeexplore.ieee.org/document/10720758

Uses

This model can be directly used for improving ASR systems in Portuguese, particularly in scenarios with limited real-world data or unique linguistic characteristics.

Out-of-Scope Use

The model is not suitable for tasks involving languages other than Portuguese without additional fine-tuning and data adjustments.

Bias, Risks, and Limitations

Users should be aware of potential biases introduced by synthetic data and ensure the quality of the data aligns with the target application's requirements. It is recommended to evaluate the model's performance on diverse datasets to identify and mitigate biases.

How to Get Started with the Model

from transformers import pipeline

model = pipeline("automatic-speech-recognition", model="my-north-ai/whisper-large-v3-pt")
result = model("path_to_audio_file.wav")
print(result)

Training Details

Training Data

The training data includes 140 hours of synthetically generated Portuguese speech and transcripts, along with real data from the Multilingual LibriSpeech Corpus (MLS), Common Voice (CV) 16.1, and the Perfil Sociolinguístico da Fala Bracarense (PSFB) dataset

Training Procedure

The model was fine tuned using DDP methodolgy across 4 A10g GPUS

Preprocessin

The preprocessing steps include text normalization, removal of special characters, and ensuring consistent formatting for TTS generation.

Training Hyperparameters

• Training regime: fp16 mixed precision
• Learning Rate: 1e-5
• Batch Size: 32
• Epochs 3

Evaluation

Testing Data, Factors & Metrics

Testing Data

The testing data includes subsets from the FLEURS dataset and PSFB, chosen for their linguistic diversity and unique speech patterns.

Metrics

Evaluation Results

Word Error Rate (WER) Comparison

WER results for FLEURS for the fine-tuned model versus pretrained model with and without text normalization.

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA A10G
• Hours used: 15
• Cloud Provider: AWS
• Compute Region: US EAST