whisper-medium-afrispeech

whisper-medium-afrispeech

Fine-tune of openai/whisper-medium on AfriSpeech-200 for African-accented English speech recognition, spanning both general and clinical/medical domains.

• Base model: openai/whisper-medium (769M)
• Training data: Professor/afrispeech-200-16khz — a clean 16 kHz mono build of AfriSpeech-200 (Intron Health): ~200 h, 120 accents, 13 countries, clinical + general.
• Language: English (African accents)

Results (test split, 6,178 clips)

Fine-tuned vs. zero-shot baseline

Zero-shot = openai/whisper-medium evaluated on the same test set with no fine-tuning.

Fine-tuning more than halves WER across the board.

WER by domain

Clinical speech remains ~2× harder than general — medical terminology (drug names, conditions, dosages) drives most of the error, so domain matters when reporting WER.

WER by accent (accents with n ≥ 25)

Performance varies widely across accents (~7% to ~45%):

The hardest accents are mostly smaller, under-represented ones — an important coverage/equity consideration. Full per-accent numbers: eval_wer_breakdown.csv.

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("Professor/whisper-medium-afrispeech")
processor = WhisperProcessor.from_pretrained("Professor/whisper-medium-afrispeech")

# audio: a 16 kHz mono waveform (numpy array)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features, language="english", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Training

• Objective: full fine-tune (all parameters), 3 epochs
• Precision / hardware: bf16 + gradient checkpointing on a single NVIDIA A40 (48 GB)
• Batch: 32 effective (16 × 2 grad-accum) · LR: 1e-5 (500 warmup, linear decay) · best model by eval WER
• Data handling: on-the-fly log-Mel feature extraction; clips > 30 s filtered out; labels truncated to Whisper's 448-token decoder cap.

Limitations & biases

• Clinical error rate is still elevated (~27%) — verify medical terms; not for unsupervised clinical use.
• Large accent disparity (~7–45% WER) — weaker on under-represented accents.
• ≤ 30 s audio; English only; inherits Whisper's general biases.

Dataset, license & attribution

Trained on AfriSpeech-200 by Intron Health, released under CC-BY-NC-SA-4.0 (non-commercial, share-alike, attribution). This derivative carries the same license.

@article{olatunji2023afrispeech,
  title={AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR},
  author={Olatunji, Tobi and others},
  journal={Transactions of the Association for Computational Linguistics},
  year={2023}
}