whisper-small-gujarati-talpada

Whisper Small — Gujarati + Talpada (તળપદા)

A fine-tuned Whisper Small ASR model for Gujarati (gu), trained so it also recognizes Talpada — the regional/village dialect of Gujarati — which vanilla Whisper frequently misses.

Model is saved in fp16 (~484 MB).

How it was trained

Base model: openai/whisper-small (244M params)

Dataset: shunyalabs/gujarati-speech-dataset — a large collection of Gujarati speech + transcripts, streamed from Hugging Face as parquet shards.

• Data: 5 parquet shards, ~2,000 samples (10-second audio windows)
• Eval: 30 held-out samples

Fine-tuning strategy (memory-efficient, runs on 16 GB GPU / Colab free tier):

• Encoder: fully frozen
• Decoder: only self-attention, layer-norms, and token embeddings trained (~65M of 244M params); cross-attention and FFN frozen → tiny optimizer, low VRAM
• Batch size 2 × gradient accumulation 8 (effective 16), learning rate 5e-5, warmup 100 steps, 3 epochs (372 steps), fp16, gradient checkpointing
• Transformers 4.57.6

Result: training loss dropped 1.65 → 0.232.

Performance

Measured on 30 held-out samples from the same dataset:

⚠️ Read the WER carefully: during eval, input audio was truncated to 10 seconds while the reference transcripts were the full sentences (up to ~25s of speech). The model could not transcribe words it never heard, which inflates WER substantially. Qualitatively the model produces phonetically accurate Gujarati — most errors are minor spelling variants (e.g. ક્રાઉન ઑફિસ → ક્રાઉન ઑફેસ), not missed words.

Comparison to other models: no strict apples-to-apples baseline was run on the same held-out set. Gujarati is a weak language for vanilla Whisper, and this fine-tune is a domain adaptation for Gujarati/Talpada speech, not a state-of-the-art benchmark model. Expect it to clearly beat stock whisper-small on spoken Gujarati, while large models (whisper-large-v3) will still win on formal/news audio — this model's edge is dialect words and low-resource robustness.

Limitations

• Trained on a small subset (~2,000 samples, 3 epochs) — coverage of rare Talpada words is limited to the training data
• Not trained on custom Talpada audio yet; the dialect vocabulary is learned from standard Gujarati text only
• Performs best on short phrases and conversational speech; dense news sentences with English loanwords remain hard
• fp16 weights: load on GPU/MPS, or convert to fp32 for CPU inference

Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

processor = WhisperProcessor.from_pretrained("princetunes/whisper-small-gujarati-talpada")
model = WhisperForConditionalGeneration.from_pretrained("princetunes/whisper-small-gujarati-talpada")
model.to("cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu"))
model.eval()

import librosa
audio, sr = librosa.load("your_audio.wav", sr=16000, mono=True)
feats = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(model.device)
forced = processor.get_decoder_prompt_ids(language="gujarati", task="transcribe")
with torch.no_grad():
    ids = model.generate(input_features=feats, forced_decoder_ids=forced, max_length=225)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Training code

Training was done in a Colab notebook (streamed data loading, frozen-encoder decoder fine-tune). See the shunyalabs/gujarati-speech-dataset repo for the raw data; contact the model author for the notebook.