whisper-small-balti

🏔️ Whisper Small — Balti ASR

State-of-the-art Automatic Speech Recognition model for Balti (بلتی), fine-tuned from openai/whisper-small on Perso-Arabic script audio data.

🏆 Benchmark Summary

• Validation WER: 17.40% (SOTA for Balti ASR)
• Previous Baseline: 26.74% (BaltiVoice) / 22.82% (Wav2Vec2)

👉 For full system details, streaming implementation, and engineering insights, visit the Balti Tarjuman GitHub Repository or read docs/Engineering Case Study.md.

---

⚡ Quick Start

import torch
import soundfile as sf
from transformers import WhisperForConditionalGeneration, WhisperProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
repo = "YuvrajGujari/whisper-small-balti"

processor = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo).to(device)

audio, _ = sf.read("balti_sample.wav")
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(inputs, forced_decoder_ids=None)

text = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcription:", text)

### 📊 Benchmark Comparison

| Model | Setup | Strategy | Validation WER |
| --- | --- | --- | --- |
| **Whisper-small (Champion)** 🏆 | **Cold-Start** | **SpecAugment + $1\times 10^{-4}$ LR** | **17.40%** |
| Wav2Vec2 XLS-R 300M | Cold-Start | Standard CTC | 22.82% |
| *BaltiVoice Literature* | — | Published Baseline | *26.74%* |
| Whisper-small (Round 2) | Warm-Start | Standard Fine-Tuning ($1\times 10^{-5}$ LR) | 36.38% |
| Whisper-small (Base) | Zero-Shot | Out-of-the-Box | 63.42% |

---

### 📈 Training Step Log

| Step | Training Loss | Validation Loss | Validation WER |
| --- | --- | --- | --- |
| 500 | 0.9031 | 0.3995 | 39.72% |
| 1000 | 0.5979 | 0.3019 | 29.23% |
| 1500 | 0.3326 | 0.2518 | 23.81% |
| 2000 | 0.1582 | 0.2235 | 20.27% |
| **2500** | **0.0609** | **0.2142** | **17.40%** |

🛠️ Key Training Highlights
Cold-Start Fine-Tuning: Reset weights to base openai/whisper-small to avoid local minima.

SpecAugment: Applied time and frequency masking (prob=0.05) to prevent overfitting.

LR & Schedule: Trained for 2,500 steps with lr=1e-4 and 500 warmup steps.