WhisperLv3-PT

WhisperLv3-PT

[!NOTE] This repository contains the <a href = "https://huggingface.co/inesc-id/WhisperLv3-X-PT-All"> WhisperLv3-X-PT-All</a> model with the <b> generation_config </b> file updated according to the <a href = "https://huggingface.co/openai/whisper-large-v3"> whisper-large-v3</a> model. This update is necessary to avoid errors in certain customization parameters, such as beam_search, task, language, etc. This rep also contains WhisperLv3-X-PT-All quantized to 4 Bits and 8 Bits using BitsAndBytes. <br> <b> llama.cpp does not support quantization for ASR models. </b>

Description

This model is a fine-tuned WhisperX variant of openai/whisper-large-v3, trained for Portuguese (pt) (European Portuguese, Brazilian and African and Asian Portuguese varieties) automatic speech recognition (ASR). From CAMÕES work.

Model Description

• Base model: openai/whisper-large-v3
• Architecture: Transformer encoder–decoder
• Training: Fine-tuned on around 800 hours of Portuguese speech
• Task: Transcription (task="transcribe")
• Compute type: float16 (recommended)

Quantization Files

This model was quantized using BitsAndBytes, allowing for a drastic reduction in memory, which enables inference on more modest systems.

Inference

If you want to use Hugging Face transformers to generate text, you can do something like this.

from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from transformes.audio_utils import load_audio

MODEL = "EliezerCarvalho/WhisperLv3-PT"

PROCESSOR_ASR = AutoProcessor.from_pretrained (MODEL)
MODEL_ASR = AutoModelForSpeechSeq2Seq.from_pretrained (MODEL)

WAV = load_audio (path, sampling_rate = PROCESSOR_ASR.feature_extractor.sampling_rate)

input = PROCESSOR_ASR (wav, sampling_rate = PROCESSOR_ASR.feature_extractor.sampling_rate, return_tensors = "pt")

inference = MODEL_ASR.generate (input["input_features"], return_timestamps = True, task = "transcribe", language = "pt", return_segments = True)