whisper-bangla-asr

Bangla & Banglish Whisper ASR — Inference Pipeline

An inference and evaluation pipeline for Bangla and Banglish (code-mixed Bangla-English) automatic speech recognition, built on top of bengaliAI/tugstugi_bengaliai-asr_whisper-medium.

---

What this repository provides

• run.py — a single script that transcribes any .flac/.wav audio file and evaluates WER/CER against a local labeled dataset. No terminal commands needed beyond running the file.
• Silence-aware chunking — audio is split at natural pause points rather than fixed intervals, reducing word-boundary errors and hallucination on silent segments.
• Local dataset evaluation — evaluates directly against your local asr_bengali files without requiring an internet download of the dataset.
• Banglish support — the base model handles code-mixed Bangla-English speech. A labeled Banglish benchmark is in progress (see Roadmap).

---

Model details

---

Evaluation

Evaluated on a local subset of the bengaliAI/asr_bengali dataset using jiwer via run.py.

Note: The WER/CER above are for the underlying base model evaluated using this pipeline's silence-aware chunking on a 10-example subset of asr_bengali. A Banglish-specific benchmark is in progress — see Roadmap.

---

Usage

Quick start

Edit the CONFIG section at the top of run.py:

AUDIO_FILE           = r"path\to\your_audio.flac"
ASR_BENGALI_DATA_DIR = r"path\to\asr_bengali\data"
REFERENCES_TSV       = r"path\to\asr_bengali\utt_spk_text.tsv"
MODEL_ID             = "bengaliAI/tugstugi_bengaliai-asr_whisper-medium"
CACHE_DIR            = r"path\to\cache"
EVAL_LIMIT           = 10

Then simply run the file. It will:

1. Auto-install all required packages on first run
2. Download model weights automatically (~3 GB, once only)
3. Transcribe your audio file and save result to transcription_result.txt
4. Evaluate WER/CER and save per-example breakdown to evaluation_results.csv

Using with transformers directly

import torch
import librosa
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

model_id = "bengaliAI/tugstugi_bengaliai-asr_whisper-medium"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)

audio, sr = librosa.load("your_audio.flac", sr=16000)
inputs = processor(audio, sampling_rate=sr, return_tensors="pt")

with torch.no_grad():
    predicted_ids = model.generate(
        inputs.input_features,
        language="bn",
        task="transcribe",
    )

transcript = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcript)

For audio longer than 30 seconds, use run.py which handles silence-aware chunking automatically.

---

Roadmap

• Labeled Banglish test set (code-mixed Bangla-English audio with ground-truth transcripts) — in progress
• WER/CER benchmark on Banglish data
• Fine-tuned model weights trained specifically on Banglish data
• Larger asr_bengali evaluation (full test split)

Contributions welcome. If you have labeled Banglish recordings, please open an issue or pull request — this is an underserved gap in Bengali ASR and community contributions would be valuable.

---

Limitations

• Banglish not yet benchmarked: Banglish support has been manually verified but not yet measured with WER/CER. A labeled test set is in progress.
• Small evaluation set: Current benchmark uses 10 examples. Larger evaluation is planned.
• Regional dialects: Performance on dialects beyond standard Bangla has not been evaluated.
• Long silences: Like all Whisper models, near-silent audio segments can produce hallucinated text. run.py includes silence detection to reduce this.
• CPU speed: On CPU-only machines, transcribing 1 minute of audio takes approximately 3–5 minutes.

---

License

MIT License — consistent with openai/whisper-medium and bengaliAI/tugstugi_bengaliai-asr_whisper-medium. See LICENSE for full text. Please retain attribution to OpenAI Whisper and the bengaliAI base checkpoint when redistributing.

---

Acknowledgements

• bengaliAI/tugstugi_bengaliai-asr_whisper-medium — base model this pipeline is built on
• openai/whisper-medium — original Whisper architecture