Downloads · 30 days
13
3% of all-time downloads
yuriyvnv/whisper-large-v3-cv-capes-filtered-pt
whisper-large-v3-cv-capes-filtered-pt is a automatic speech recognition model from yuriyvnv. Use it when you need speech turned into text. It is set up for transformers. The card lists the license as apache-2.0.
This model is a fine-tuned version of openai/whisper-large-v3 for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with the CAPES synthetic dataset filtered using…
Downloads · 30 days
13
3% of all-time downloads
All-time downloads
443
Public
Parameters
1.5B
204 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors6.2 GB · 100%
From the Hugging Face model README
This model is a fine-tuned version of openai/whisper-large-v3 for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with the CAPES synthetic dataset filtered using WAVe (Word-Aligned Verification), achieving dramatic improvements over the baseline CAPES filtering approach.
This model demonstrates the superiority of word-level filtering (WAVe) over sentence-level filtering by applying our new methodology to the established CAPES dataset. It directly compares with:
By applying WAVe filtering to the same CAPES dataset, this model achieves:
This demonstrates that quality filtering is more important than data quantity for robust ASR performance.
CAPES Dataset Characteristics:
WAVe's Advantage:
| Property | Value |
|---|---|
| Base Model | openai/whisper-large-v3 |
| Language | Portuguese (pt) |
| Task | Automatic Speech Recognition (transcribe) |
| Parameters | 1550M |
| Training Data | Common Voice 17.0 + CAPES Filtered (WAVe Word-Level) |
| Total Training Samples | ~45,000 |
| Sampling Rate | 16kHz |
| Filtering Method | WAVe word-level (q ≥ 0.8) |
| Metric | Value |
|---|---|
| Validation Loss | 0.1055 |
| Validation WER | 7.38% |
| Test WER (Common Voice) | 7.95% |
| Test WER (MLS) | 6.89% |
| Best Checkpoint | Step 300 |
| Max Training Steps | 880 |
| Model | Filtering Method | Synthetic Samples | Max Steps | Test WER (CV) | Test WER (MLS) | MLS Improvement |
|---|---|---|---|---|---|---|
| my-north-ai/whisper-large-v3-pt | Sentence-level | ~33k | ~1,080 | ~8.4% | ~13.5% | Baseline |
| CAPES Baseline | Sentence-level | 33.2k | 1,080 | 8.43% | 13.54% | Baseline |
| This Model (WAVe) | Word-level | 23k | 880 | 7.95% | 6.89% | +49% |
| Model | Dataset | Filtering | Test WER (CV) | Test WER (MLS) | Best For |
|---|---|---|---|---|---|
| CV Only | Common Voice | None | 11.78% | 15.31% | Baseline |
| High-Quality | Our Synthetic | q ≥ 0.8 | 7.94% | 12.41% | In-domain |
| Mixed | Our Synthetic | q ≥ 0.5 | 8.33% | 10.27% | Balanced |
| CAPES Baseline | CAPES | Sentence-level | 8.43% | 13.54% | State-of-the-art replication |
| CAPES WAVe (this) | CAPES | Word-level | 7.95% | 6.89% | Cross-domain champion |
| Source | Samples | Description |
|---|---|---|
| Common Voice 17.0 Portuguese | 21,866 | Real crowdsourced speech |
| CAPES Filtered (WAVe q ≥ 0.8) | ~23,000 | Academic thesis-derived synthetic speech with word-level filtering |
| Total | ~45,000 |
By applying WAVe's word-level filtering (q ≥ 0.8) to the CAPES dataset:
| Parameter | Value |
|---|---|
| Learning Rate | 5e-6 |
| Batch Size (Global) | 256 |
| Warmup Steps | 200 |
| Max Epochs | 5 |
| Precision | BF16 |
| Optimizer | AdamW (fused) |
| Eval Steps | 50 |
| Metric for Best Model | eval_loss |
from transformers import pipeline
transcriber = pipeline(
"automatic-speech-recognition",
model="yuriyvnv/whisper-large-v3-cv-capes-filtered-pt",
device="cuda"
)
result = transcriber("path/to/portuguese_audio.wav")
print(result["text"])
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-cv-capes-filtered-pt")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-cv-capes-filtered-pt")
model.to("cuda")
audio, sr = librosa.load("path/to/portuguese_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")
predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)
model.generation_config.language = "pt"
model.generation_config.task = "transcribe"
WAVe (Word-Aligned Verification) achieves superior performance through:
Why It Matters for CAPES:
This model is ideal when:
This model proves a fundamental principle in synthetic speech augmentation:
Word-level quality filtering is more effective than sentence-level filtering for ASR training, especially with longer utterances.
The 49% improvement in cross-domain generalization while using 30% less data demonstrates that:
This model demonstrates WAVe (Word-Aligned Verification) filtering applied to the CAPES dataset. While the WAVe methodology paper is currently under review, the CAPES dataset and sentence-level filtering baseline are from our previous work:
@article{perezhohin2024enhancing,
title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
journal={IEEE Access},
year={2024},
publisher={IEEE}
}
@article{perezhohin2026wave,
title={WAVe: Word-aligned verification of synthetic speech for ASR},
author={Perezhohin, Yuriy and Castelli, Mauro},
journal={Information Sciences},
pages={123591},
year={2026},
publisher={Elsevier}
}
Apache 2.0