whisper-small-konkani-v2

Whisper Small — Konkani ASR (LoRA)

A LoRA (PEFT) fine-tuned adapter for openai/whisper-small, targeting automatic speech recognition (ASR) for Konkani, with an initial focus on agricultural speech and extended training on general Konkani speech and spoken numbers (1–100).

Model Details

• Base model: openai/whisper-small
• Fine-tuning method: LoRA (PEFT) — parameter-efficient fine-tuning suited to low-resource languages and limited compute
• Language: Konkani (kok) — Whisper has no native language token for Konkani; transcription is typically run with language="mr" (Marathi) as the closest supported script/phonetic match, or with language auto-detection
• Training hardware: Google Colab, free-tier NVIDIA T4 GPU

Training Data

1. ILCI Agriculture Speech Corpus (VidyaapatiGU-Internship/ILCI_Agriculture_Speech_Corpus) — ~1,750 paired audio/transcript samples, ~1.7 hours of audio, agricultural domain, split 80/10/10 (train/val/test)
2. IndicVoices-R (ai4bharat/indicvoices_r, Konkani split) — broader general-domain Konkani speech, used to extend the initial adapter's coverage beyond agriculture
3. Konkani numbers (1–100) — supplementary recorded speech samples for numeral recognition

Training Procedure

• Base adapter trained on the ILCI Agriculture corpus using LoRA/PEFT
• Adapter published, then further trained on IndicVoices-R (Konkani) and Konkani number recordings, continuing from the saved adapter weights (loaded via PeftModel.from_pretrained, saved via PeftModel.save_pretrained — adapter-only checkpoints, not full Trainer checkpoints)

Evaluation Results

Note: WER/CER remain relatively high, consistent with the low-resource nature of Konkani ASR and the limited training data volume. This adapter should be treated as an early-stage research checkpoint rather than a production-ready transcription system.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
import torch, librosa

base_model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(base_model, "sandeepsawant28/whisper-small-konkani-v2")
model = model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

processor = WhisperProcessor.from_pretrained("openai/whisper-small")

audio, sr = librosa.load("your_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(model.device)

with torch.no_grad():
    predicted_ids = model.generate(input_features, language="mr", task="transcribe")

print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0])

Limitations

• Whisper lacks a native Konkani language token; results depend on the workaround language code used (mr or auto-detect)
• Trained on a relatively small dataset (~1.7 hrs core corpus); WER/CER indicate room for improvement with more data and training time
• Evaluated primarily on agriculture-domain and numeral speech; general conversational Konkani performance is less tested

Credits

Developed as part of the Vidyaapati Goa University Student Internship (VidyaapatiGU-Internship).