whisper-small-ar-fleurs

Whisper Small — Arabic (fine-tuned on FLEURS ar_eg)

openai/whisper-small fine-tuned on the Arabic (ar_eg) subset of Google FLEURS (~6 hours of read MSA speech by Egyptian-accented speakers). Fine-tuning reduces normalized WER on the held-out FLEURS test split from 28.0% → 19.9% (−29% relative), at unchanged inference cost.

Results (FLEURS ar_eg test, 427 samples, corpus-level)

Normalization: NFC, diacritics/tatweel removal, alef & ya unification, punctuation stripping — applied identically to references and predictions. One undecodable test sample excluded (documented).

Out-of-domain: spontaneous Egyptian (Casablanca)

Evaluated zero-shot on a 400-clip subset of Casablanca Egypt test (seed 42) — real spontaneous Egyptian dialect, never seen in training. All clips ≤26s (duration-audited: no truncation).

Three takeaways: (1) no off-the-shelf model — including turbo — reaches usable accuracy on spontaneous Egyptian; (2) fine-tuning on read MSA still transferred a 21-point WER gain, driven almost entirely by decoding discipline (insertions collapsed 0.33 → 0.07 — the lowest hallucination rate of the three, below even turbo); (3) dialect WER partly reflects non-standardized dialect orthography — CER is the kinder acoustic view. Full analysis: phase 4 report.

Training

• Data: FLEURS ar_eg train (2,102 clips ≤30s ≈ 6h); labels = dataset's transcription field
• 5 epochs, lr 1e-5 (warmup 50), fp16, effective batch 16, gradient checkpointing, seed 42
• Best checkpoint selected by validation WER (epoch 4, val WER 19.33% — consistent with test 19.90%)

Error analysis (worst-50 rematch vs the zero-shot baseline)

Acoustic substitutions improved most (16/19 samples), numbers 7/7, named entities 9/12; rare technical terms unchanged. Number-format gains partly reflect adaptation to FLEURS's orthographic conventions and may not transfer out of domain.

Usage

# pip install transformers torch librosa soundfile

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

# 1. Load the model
MODEL_ID = "zainab11/whisper-small-ar-fleurs"
processor = WhisperProcessor.from_pretrained(MODEL_ID)
model = WhisperForConditionalGeneration.from_pretrained(MODEL_ID)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device).eval()

# 2. Load any audio file (WAV/MP3/...) — converted to mono 16 kHz automatically
audio_path = "my_audio.wav"          # <-- put your file here
audio, sr = librosa.load(audio_path, sr=16000, mono=True)
print(f"Audio duration: {len(audio)/sr:.2f}s")

# 3. Audio -> log-Mel features (+ attention mask)
inputs = processor(audio, sampling_rate=16000,
                   return_attention_mask=True, return_tensors="pt")
input_features = inputs.input_features.to(device)
attention_mask = inputs.attention_mask.to(device)

# 4. Transcribe (Arabic)
with torch.no_grad():
    ids = model.generate(input_features,
                         attention_mask=attention_mask,
                         language="ar", task="transcribe")

# 5. Decode
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Note: clips longer than 30s are truncated by this simple example (Whisper's window). For long audio, use the chunked pipeline or segment first.

Intended use & limitations

Fine-tuned on read MSA speech with Egyptian-accented speakers — not spontaneous Egyptian dialect, noisy telephony, or code-switched speech. Inherits Whisper's known long-form repetition/hallucination behavior. As measured on dialectal data above, the model is not suitable for spontaneous Egyptian without further dialect fine-tuning

Part of an end-to-end Arabic ASR study (baselines → normalization → error analysis → fine-tuning): github.com/zainabayman11/whisper-ar-skeleton

🎙️ Try it live: whisper-ar-demo