whisper-small-high-mixed-nl

Whisper-Small Dutch - High-Quality Filtered Synthetic Data

This model is a fine-tuned version of openai/whisper-small for Dutch automatic speech recognition (ASR). It was trained on Common Voice 17.0 Dutch combined with WAVe-filtered synthetic speech data using a strict high-quality threshold (q ≥ 0.8).

Introduction

How the Data Was Created

The training data combines real speech from Common Voice 17.0 with synthetic speech generated through a two-stage pipeline:

1. Transcript Generation: We used GPT-4o-mini to generate Dutch transcripts that match the word count distribution observed in Common Voice, ensuring realistic utterance lengths and diverse linguistic content.

2. Speech Synthesis: Each transcript was converted to audio using OpenAI's TTS-1 model with 9 different voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer), producing 34,898 synthetic samples.

3. Quality Filtering with WAVe: Raw synthetic speech often contains defects such as mispronunciations, omitted words, or prosodic anomalies. To address this, we applied WAVe (Word-Aligned Verification), a model that assesses audio-text alignment at the word level rather than the sentence level. WAVe uses multi-head attention to align each word to its corresponding audio frames and assigns per-word confidence scores via a GLU-based scorer. For this model, only samples scoring above the strict threshold (q ≥ 0.8) were retained, resulting in 10,555 high-quality synthetic samples.

How the Model Was Created

The model was fine-tuned from openai/whisper-small using the Hugging Face Transformers library with the following approach:

1. Mixed Training: Combined 34,952 real speech samples from Common Voice 17.0 Dutch with 10,555 strictly WAVe-filtered synthetic samples (45,507 total).

2. Optimization: Trained for 5 epochs with a learning rate of 1e-5, global batch size of 256, and BF16 precision on an NVIDIA H200 GPU.

3. Checkpoint Selection: The best checkpoint was selected based on validation loss, occurring at step 350 with a validation loss of 0.1493.

This high-quality filtering approach achieves 35% reduction in training steps compared to using all synthetic data, while maintaining competitive ASR performance.

Model Details

Evaluation Results

This Model (whisper-small-high-mixed-nl)

Comparison with Other Training Configurations (Whisper-Small Dutch)

Key Performance Highlights

• Most efficient training: Only 890 max steps (35% fewer than unfiltered)
• 1.2% relative improvement on Common Voice test set vs baseline (11.00% vs 11.13%)
• 2.6% relative improvement on MLS benchmark vs baseline (29.91% vs 30.71%)
• Best quality-to-compute ratio: Achieves strong results with minimal synthetic data

Training Data

Dataset Composition

Synthetic Data Generation Pipeline

The synthetic dataset (yuriyvnv/synthetic_transcript_nl) was generated using:

1. Transcript Generation: GPT-4o-mini, matching Common Voice word count distribution
2. Speech Synthesis: OpenAI TTS-1 model with 9 voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer)
3. Quality Filtering: WAVe model with strict threshold q ≥ 0.8

WAVe Quality Distribution (Dutch Synthetic Data)

Training Procedure

Hyperparameters

Training Infrastructure

• GPU: NVIDIA H200 (140GB VRAM)
• Operating System: Ubuntu 22.04
• Framework: Hugging Face Transformers

Training Curve

Step  100: val_loss = 0.1824
Step  200: val_loss = 0.1607
Step  300: val_loss = 0.1520
Step  350: val_loss = 0.1493 ← Best checkpoint
Step  500: val_loss = 0.1498
Step  700: val_loss = 0.1514
Step  850: val_loss = 0.1538

Usage

Transcription Pipeline

from transformers import pipeline

transcriber = pipeline(
    "automatic-speech-recognition",
    model="yuriyvnv/whisper-small-high-mixed-nl",
    device="cuda"
)

result = transcriber("path/to/dutch_audio.wav")
print(result["text"])

Direct Model Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-small-high-mixed-nl")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-small-high-mixed-nl")
model.to("cuda")

audio, sr = librosa.load("path/to/dutch_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Specifying Language

model.generation_config.language = "nl"
model.generation_config.task = "transcribe"

Methodology

This model leverages WAVe (Word-Aligned Verification), a word-level quality assessment method for filtering synthetic speech data. Unlike sentence-level filtering approaches, WAVe:

• Aligns each word to its corresponding audio frames using multi-head attention
• Assigns per-word confidence scores via a GLU-based scorer
• Detects localized synthesis errors (mispronunciations, omitted words, prosodic anomalies)
• Achieves 6.5% improvement over sentence-level filtering methods

The strict threshold (q ≥ 0.8) retains only the top 30.2% of synthetic samples, prioritizing quality over quantity for maximum training efficiency.

When to Use This Model

This model is ideal when:

• Compute resources are limited: 35% fewer training steps than unfiltered approaches
• Quick fine-tuning is needed: Smaller dataset enables faster iteration
• Good balance of accuracy and efficiency: Competitive performance with minimal compute

Consider other variants based on your needs:

• whisper-small-mixed-nl: Better absolute performance, more compute
• whisper-small-cv-only-nl: Baseline without synthetic data

Limitations

• Domain specificity: Optimized for general Dutch; may underperform on technical domains
• Acoustic conditions: Trained on clean speech; noise robustness not guaranteed
• Dialect coverage: Performance may vary across Dutch regional variants

Citation

@article{perezhohin2024enhancing,
  title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
  author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
  journal={IEEE Access},
  year={2024},
  publisher={IEEE}
}
@article{perezhohin2026wave,
  title={WAVe: Word-aligned verification of synthetic speech for ASR},
  author={Perezhohin, Yuriy and Castelli, Mauro},
  journal={Information Sciences},
  pages={123591},
  year={2026},
  publisher={Elsevier}
}

References

• Base Model: openai/whisper-small
• Training Data (Real): mozilla-foundation/common_voice_17_0
• Training Data (Synthetic): yuriyvnv/synthetic_transcript_nl
• Whisper Paper: Robust Speech Recognition via Large-Scale Weak Supervision
• IEEE Access Paper: Enhancing ASR with Semantic Audio Filtering

License

Apache 2.0