nepali-asr-whisper-medium

Whisper-Medium (Nepali)

Fine-tuned multilingual pre-trained ASR for Nepali — part of the controlled six-model Nepali ASR Benchmark released alongside the paper Comparative Analysis of Multilingual Pre-trained Models for Nepali Automatic Speech Recognition.

Model summary

Benchmark results

Word and character error rate on the three independent test sets, plus inference Real-Time Factor (RTF) on a single NVIDIA L4 with batch size 1 (lower RTF = faster).

WER and CER are computed with jiwer after NFC normalisation of both reference and hypothesis. Best in-domain performance during training was WER 15.12% / CER 3.81%.

Note on Common Voice (Nepali). As of 2025, Mozilla distributes Common Voice via the Mozilla Data Collective (MDC) instead of the Hugging Face Hub. The benchmark numbers below were produced on the ne-NP test split of CV 25.0 downloaded from MDC.

Usage

# pip install transformers librosa soundfile torch
import torch
import librosa
import unicodedata
from transformers import WhisperForConditionalGeneration, AutoProcessor

REPO = "sumanpaudel1997/nepali-asr-whisper-medium"
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")

processor = AutoProcessor.from_pretrained(REPO)
model = WhisperForConditionalGeneration.from_pretrained(REPO).to(device).eval()

# Load and resample to 16 kHz mono (the training rate).
audio, _ = librosa.load("your_clip.wav", sr=16000, mono=True)

inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
forced_decoder_ids = processor.get_decoder_prompt_ids(language="ne", task="transcribe")
with torch.no_grad():
    generated = model.generate(
        inputs.input_features.to(device),
        forced_decoder_ids=forced_decoder_ids,
        max_new_tokens=256,
    )

text = processor.batch_decode(generated, skip_special_tokens=True)[0]
text = unicodedata.normalize("NFC", text).strip()
print(text)
# Example output (depends on clip): 'नेपालको संविधानले सबै नागरिकलाई समान अधिकार प्रदान गर्दछ।'

Training protocol (summary)

• Audio resampled to 16 kHz mono, transcriptions Unicode-normalised to NFC.
• Utterances shorter than 0.5 s or longer than 30 s removed.
• Family-matched optimiser/scheduler: AdamW, family-specific learning rate, warmup, and decay.
• SpecAugment applied to every model.
• Early stopping when validation WER stagnated for three consecutive evaluation rounds.

Full hyperparameters and the controlled comparison across all six models are in the paper (see Citation).

Limitations

• Fine-tuned on read speech only. Performance on spontaneous conversational speech, code-switched (Nepali–English / Hindi) speech, and dialectal variation is out of scope for this release.
• No external language model is used during decoding — reported numbers reflect acoustic-model performance only. Shallow fusion / n-best rescoring with a Nepali LM is expected to deliver additional gains, especially on noisy audio.
• Crowd-sourced Common Voice (~48 % WER for the best model in the suite) remains the largest open problem; this checkpoint is no exception.

License

Released under CC-BY-NC-4.0 — research/non-commercial use, attribution required. The licences of the source base model (linked above) and of the fine-tuning datasets (OpenSLR SLR54, FLEURS, Common Voice) apply on top of this restriction. For commercial use, contact the authors.

Acknowledgements

M.Sc. (Data Science) thesis carried out at the School of Mathematical Sciences, Institute of Science and Technology, Tribhuvan University, Kathmandu, Nepal, under the supervision of Asst. Prof. Sarbin Sayami (Central Department of Computer Science and Information Technology). Thanks to the open-source teams behind Wav2Vec 2.0, Whisper, MMS, IndicWav2Vec, and NVIDIA NeMo for the publicly released base checkpoints, and to OpenSLR, Google FLEURS, and Mozilla Common Voice for the source corpora.