waxal-whisper-small-spec_aug_v1

waxal-whisper-small-spec_aug_v1

Fine-tuned openai/whisper-small on Google WaxalNLP (Lingala, Shona, Luganda) with SpecAugment on log-mel features.

Training champion from the whisper-small exploration (Phase-1 holdout score 0.2670 = 0.5·WER + 0.5·CER). Continued / decode-focused variants (e.g. freeze-encoder + higher LR) were explored on top of this recipe; see the sibling base card mmwanje/waxal-whisper-small.

Training notes

• Joint multilingual FT (no language tokens in labels — better for Phase 2)
• SpecAugment applied on padded mel batches at train time
• 2×P100 DDP; typical recipe shape: batch=2, grad_accum=4, ~800 steps (see repo experiment log)
• Base init: openai/whisper-small (or continue from mmwanje/waxal-whisper-small)

Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration

repo = "mmwanje/waxal-whisper-small-spec_aug_v1"
processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo)

Phase 2

Unlabeled clips: run MMS LID (facebook/mms-lid-4017) → force Whisper into {lingala, shona, swahili}. Decode sweeps (no_cond, lang prompts, beams) are inference-only — no retrain required.