whisper-large-v2-nepali-financial

Model Card for Whisper Large-v2 + LoRA (Nepali Financial)

This is a domain-adapted version of Whisper large-v2 fine-tuned with LoRA on the NepFinSpeech-403 dataset for low-resource Nepali financial speech recognition. It is designed to recognize spoken commands for mobile wallet operations (send, load, balance) and handles Devanagari numerals.

The model significantly reduces word error rate (WER) on financial utterances compared to the zero-shot baseline, and is deployed in the SpeakPay voice-first e-wallet application.

Performance

Evaluation on the held-out test set of NepFinSpeech-403:

Relative WER reduction of 67.2% over the zero-shot baseline, with improvements on 59/60 test utterances (sign test, p = 3.5×10⁻¹⁸). Transaction Success Rate improves from 1.67% to 33.33%.

Usage

This is a PEFT adapter and must be loaded together with the base model openai/whisper-large-v2. For usage examples and full training/inference pipeline, please refer to the SpeakPay GitHub repository.

Model Details

• Base model: openai/whisper-large-v2
• Fine-tuning method: LoRA (rank 32, alpha 64)
• Task: Automatic speech recognition (Nepali financial domain)
• Language: Nepali
• License: MIT (code), CC-BY 4.0 (dataset)

Model Sources

• Repository: github.com/subedibiraj/speakpay
• Paper: SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
• Demo: speakpay.biraj-subedi.com.np
• Dataset: huggingface.co/datasets/birajsubedi/NepFinSpeech

Evaluation

The model was evaluated on the held‑out test set of NepFinSpeech, containing 60 utterances spanning all command types and numeral ranges. Metrics reported are word error rate (WER), character error rate (CER), and Devanagari numeral accuracy (NumAcc). Full analysis (including confusion patterns and data‑efficiency sweeps) is available in the paper and in the analysis/ directory of the repository.

Training Procedure

The LoRA adapter was fine-tuned using the training/ pipeline from the SpeakPay repository. Hyperparameters and details can be found in the repository documentation.

Framework versions

• PEFT 0.6.2

Citation

@misc{subedi2026speakpay,
  title   = {SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for
             Low-Resource Nepali Financial Speech Recognition},
  author  = {Subedi, Biraj},
  year    = {2026},
  url     = {https://github.com/subedibiraj/speakpay}
}

More Information

• SpeakPay project page
• NepFinSpeech dataset card