whisper-large-v3-svarah-indic

Whisper-large-v3 QLoRA for Indian-Accented English (Svarah)

A QLoRA adapter for openai/whisper-large-v3 fine-tuned on Svarah from AI4Bharat. Svarah has 9.6 hours of English speech from 117 speakers across 65 districts in 19 Indian states. The adapter adapts Whisper's largest model to Indian accents. The base model was loaded in 4-bit NF4, so it trains on a single mid-range GPU.

The adapter is only ~63 MB. Load it on top of the public base model; see Usage.

Model details

Training

Loss curve

Eval loss fell 21% between steps 100 and 500 (0.255 → 0.202). After the first epoch (step 375), train loss drops sharply while eval loss flattens, so 500 steps is about where overfitting starts. The TensorBoard logs are in runs/.

Word error rate (WER) on the held-out split hasn't been measured yet. Eval loss is the only held-out metric reported here.

Usage

import torch, librosa
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
base = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "Akash-Sakala/whisper-large-v3-svarah-indic")
model = model.merge_and_unload().eval()

audio, _ = librosa.load("sample.wav", sr=16_000)
features = processor(audio, sampling_rate=16_000, return_tensors="pt").input_features
features = features.to(model.device, torch.float16)

with torch.no_grad():
    ids = model.generate(features, language="en", task="transcribe")
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Low-VRAM option: load the base model with BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4") as it was during training, and skip merge_and_unload().

Limitations

• Svarah is published as an evaluation benchmark. Because this adapter trains on part of it, don't compare scores on Svarah with published zero-shot numbers. If the split was made per clip rather than per speaker, eval speakers may also appear in training.
• English only. Hindi–English code-mixing is outside the training distribution.
• Svarah is small (~9.6 h), so the adapter mainly shifts the model toward the accent. It doesn't teach new vocabulary at scale.

Related

• Akash-Sakala/whisper-small-emotion: a LoRA for Whisper-small on Indian-English expressive speech (WER 9.16% → 4.51%)
• Akash-Sakala/qwen3.5-4b-transcript-formatter-lora: an ASR post-processor that formats raw transcripts

Citation (dataset)

@inproceedings{javed2023svarah,
  title     = {Svarah: Evaluating English ASR Systems on Indian Accents},
  author    = {Javed, Tahir and Joshi, Sakshi and Nagarajan, Vignesh and Sundaresan, Sai and Nawale, Janki and Raman, Abhigyan and Bhogale, Kaushal and Kumar, Pratyush and Khapra, Mitesh M.},
  journal   = {arXiv preprint arXiv:2305.15760},
  year      = {2023}
}