whisper-tiny-grpo

whisper-tiny-grpo

openai/whisper-tiny finetuned with GRPO (Group Relative Policy Optimization) using word error rate as the reward, across 81 Common Voice locales (79 distinct Whisper languages).

Instead of cross-entropy against a single reference, the model samples a group of candidate transcriptions per clip, scores each by its WER against the ground truth, and is nudged toward the lower-error candidates with a policy-gradient objective regularized by a KL penalty to the original model. Training code: will-rice/whisper-rl.

Results

Evaluated on the Common Voice 26.0 test split — 4,800 clips round-robin across all 81 locales in the index, greedy decoding, corpus-level rates.

Improves 69 of 81 locales.

Rates are bucketed by Common Voice locale, not by Whisper language. The two differ: zh-CN, zh-HK and zh-TW are separate locales that all prompt with the same <|zh|> token, and regional codes like pa-IN, ne-NP and sv-SE map to <|pa|>, <|ne|> and <|sv|>. 81 locales span 79 Whisper languages.

Largest gains

Baseline WER above 1.0 means the model emitted more tokens than the reference contains — runaway insertion, not merely inaccurate transcription. The largest gains are where that behaviour was worst.

Regressions (11 of 81 locales)

The regressions cluster in higher-resource languages where whisper-tiny was already reasonable. The method trades some high-resource accuracy for large low-resource gains.

Evaluation methodology

Held-out in both senses that matter:

• Disjoint from training data. This checkpoint trained on Common Voice 22; evaluation is on Common Voice 26 test. Common Voice partitions speaker-disjoint and keeps assignments stable across releases — intersecting CV22 train with CV26 test over six locales found 1 shared speaker in 4,900.
• Disjoint from checkpoint selection. Training selects checkpoints on validation reward. The maximum of a noisy sequence of validation passes is optimistically biased, so validation numbers overstate. Every figure here is from test, which nothing in training touched.

Baseline and finetuned models score the same clips in the same order — the evaluation slice is materialized with take() and never shuffled — so a difference can only come from the model.

Limitations

• Roughly 59 clips per locale at this sample count. The overall figures are solid; any single low-resource locale's rate is noisy.
• Compared only against openai/whisper-tiny, not against supervised finetuning on the same data. This shows GRPO beats the base model, not that it beats SFT.
• Three checkpoints from the same training line were scored, and this one was best on test. Selecting the best of three on the evaluation set slightly flatters this specific number — though all three beat the baseline by a similar margin (WER −0.298 to −0.349), so the improvement itself does not depend on which was picked.
• CER is the more meaningful figure for languages without word boundaries (ja, th, zh-*).

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("wrice/whisper-tiny-grpo")
processor = WhisperProcessor.from_pretrained("wrice/whisper-tiny-grpo")

The language token must be pinned at generation time; Whisper's own detection mislabels lower-resource clips, which is what the training setup assumes.