whisper-it

Whisper Small - Italian

This model is a fine-tuned version of openai/whisper-small on the Common-voice-11.0 dataset. It achieves the following results on the evaluation set:

• Loss: 0.4549
• Wer: 200.40

Model description

Whisper is a pre-trained model for automatic speech recognition (ASR) published in September 2022 by the authors Alec Radford et al. from OpenAI. Unlike many of its predecessors, such as Wav2Vec 2.0, which are pre-trained on un-labelled audio data, Whisper is pre-trained on a vast quantity of labelled audio-transcription data, 680,000 hours to be precise. This is an order of magnitude more data than the un-labelled audio data used to train Wav2Vec 2.0 (60,000 hours). What is more, 117,000 hours of this pre-training data is multilingual ASR data. This results in checkpoints that can be applied to over 96 languages, many of which are considered low-resource.

When scaled to 680,000 hours of labelled pre-training data, Whisper models demonstrate a strong ability to generalise to many datasets and domains. The pre-trained checkpoints achieve competitive results to state-of-the-art ASR systems, with near 3% word error rate (WER) on the test-clean subset of LibriSpeech ASR and a new state-of-the-art on TED-LIUM with 4.7% WER (c.f. Table 8 of the Whisper paper). The extensive multilingual ASR knowledge acquired by Whisper during pre-training can be leveraged for other low-resource languages; through fine-tuning, the pre-trained checkpoints can be adapted for specific datasets and languages to further improve upon these results.

Intended uses & limitations

This fine-tuned model goals are to experiment and to allow the authors to gain skills and knowledge on how this process is carried out. The model serves as basis for the development of a small gradio-hosted application that transcribes recordings and audio files in italian. This application also allows to insert a YouTube link of an Italian video ad gain a transciption.

The limitations of this project mainly regard the limited resources available to fine-tune the model, namely Google Colab free-version and a Google Drive used as feature storage, that had a limited space. The time dedicated to this project was also limited, as it had to fit within academic deadlines.

Training and evaluation data

The training was carried out on Google Colab platform, and the evalutation data (as the whole dataset) was taken from the Common-voice-11.0 dataset reducing the dataset to only 10% of the original dataset, to avoid the training the model for too much time.

Training procedure

The training was conducted on Google Colab, using Jupyter Notebook to write code and document the training. Google Drive was used as Feature store. Due to the limited resources of the free version of Google Colab, checkpointing was used to save partial results and then resume in a following run. The notebook was run 15 times, with approximately 40 min for each 100 steps of training for a total of 26.5h of training. Keep in mind that Google Colab was available to us for no more than 4 h a day, so around 7 days were necessary for training alone.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 16
• eval_batch_size: 8
• training_steps: 4000
• gradient_accumulation_steps: 2
• save_steps: 100
• eval_steps: 100

Training results

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.0+cu118
• Datasets 2.15.0
• Tokenizers 0.15.0