whisper-large-v3-estonian

Whisper-large-v3 Estonian

Fine-tuned OpenAI Whisper-large-v3 for Estonian automatic speech recognition, augmented with TTS-generated synthetic data.

This model is part of the paper Synthetic speech augmentation for low-resource Estonian and Slovenian ASR: Comparing Parakeet-TDT and Whisper, published in Machine Learning with Applications 26 (2026) 101043.

Model Description

• Architecture: Encoder-decoder Transformer (Whisper)
• Parameters: 1.55B
• Tokenizer: 51,865-token byte-level BPE
• Base model: openai/whisper-large-v3
• Fine-tuning data: CommonVoice 17.0 Estonian + ~5,850 synthetic sentences (LLM-generated text + OpenAI TTS)
• Training config: CV + Synth All (full synthetic corpus with quality filtering)

Evaluation Results

Raw WER/CER (no text normalization)

Normalized WER/CER (lowercase + punctuation removal)

Improvement over baselines

All improvements are statistically significant (paired bootstrap, p < 0.001, n = 100,000).

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

# Load model and processor
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-estonian")
processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-estonian")

# Load audio (16kHz mono)
import librosa
audio, sr = librosa.load("audio.wav", sr=16000)

# Transcribe
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
    predicted_ids = model.generate(input_features, language="et", task="transcribe")
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Training Details

• Optimizer: AdamW fused (lr=5e-5)
• Schedule: Linear decay with 10% warmup
• Effective batch size: 128 (64 x 2 gradient accumulation)
• Epochs: 5
• Best model: selected by eval_loss
• Precision: bf16
• Seed: 42

Synthetic Data Augmentation

The synthetic training data was generated using a three-stage pipeline:

1. Text generation: GPT-5-mini generates diverse sentences across paraphrase, domain expansion, and morphological categories
2. LLM-as-judge validation: Each sentence validated for grammaticality, naturalness, and language purity
3. Speech synthesis: OpenAI gpt-4o-mini-tts with 11-voice rotation

Dataset: yuriyvnv/synthetic_asr_et_sl

Acknowledgments

• Base model: OpenAI Whisper-large-v3
• Training data: Mozilla Common Voice 17.0
• Evaluation: Google FLEURS

Citation

@article{PEREZHOHIN2026101043,
  title   = {Synthetic speech augmentation for low-resource Estonian and Slovenian ASR: Comparing Parakeet-TDT and Whisper},
  journal = {Machine Learning with Applications},
  volume  = {26},
  pages   = {101043},
  year    = {2026},
  issn    = {2666-8270},
  doi     = {10.1016/j.mlwa.2026.101043},
  url     = {https://www.sciencedirect.com/science/article/pii/S2666827026002082},
  author  = {Yuriy Perezhohin and Mauro Castelli and Aleš Popovič},
}