Downloads · 30 days
21
7% of all-time downloads
Tachyeon/whisper-large-v3-turbo-hindi-lora
whisper-large-v3-turbo-hindi-lora is a automatic speech recognition model from Tachyeon. Use it when you need speech turned into text. It is set up for peft. The card lists the license as apache-2.0.
A LoRA fine-tuned adapter for openai/whisper-large-v3-turbo optimized for Hindi (Devanagari) speech recognition.
Downloads · 30 days
21
7% of all-time downloads
All-time downloads
294
Public
Repo size
111 MB
Likes
0
Public
Click a slice to open those files.
.safetensors111 MB · 95%
From the Hugging Face model README
A LoRA fine-tuned adapter for openai/whisper-large-v3-turbo optimized for Hindi (Devanagari) speech recognition.
| Model | WER (%) | Eval Set |
|---|---|---|
openai/whisper-large-v3-turbo (baseline) | 35.56 | FLEURS hi_in test (n=418) |
| + LoRA fine-tune (this model) | 22.25 | FLEURS hi_in test (n=418) |
| + CTranslate2 INT8 deployment | 22.70 | FLEURS hi_in test (n=418) |
37.4% relative WER reduction. INT8 deployment via faster-whisper adds only 0.45% WER degradation.
Evaluation uses Whisper-default text normalization. See Normalization Notes below.
| Model | WER (%) | Method | Training Data |
|---|---|---|---|
| collabora/whisper-large-v2-hindi | 5.33 | Full fine-tune | Multi-corpus (100h+) |
| vasista22/whisper-hindi-large-v2 | 6.80 | Full fine-tune | Multi-corpus (100h+) |
openai/whisper-large-v3-turbo | 35.56 | Zero-shot | — |
| This model (LoRA) | 22.25 | LoRA (3.33% params) | FLEURS only (~3.5h) |
Note: The collabora and vasista22 models are full fine-tunes trained on hundreds of hours of multi-corpus Hindi data. This model uses only ~3.5 hours of FLEURS data with a lightweight LoRA adapter, making it a fundamentally different trade-off: minimal data and compute for significant WER improvement over the zero-shot baseline.
| Step | Train Loss | Eval Loss | Eval WER (%) |
|---|---|---|---|
| 50 | 0.263 | 0.259 | 29.40 |
| 100 | 0.210 | 0.234 | 25.74 |
| 150 | 0.145 | 0.223 | 24.49 |
| 200 | 0.148 | 0.217 | 23.43 |
| 250 | 0.146 | 0.213 | 23.82 |
| 300 | 0.096 | 0.215 | 22.42 |
| 350 | 0.109 | 0.215 | 22.50 |
Best checkpoint: step 300 (lowest val WER). Test WER: 22.25%.
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel
BASE_MODEL = "openai/whisper-large-v3-turbo"
ADAPTER = "Tachyeon/whisper-large-v3-turbo-hindi-lora"
processor = WhisperProcessor.from_pretrained(BASE_MODEL)
base_model = WhisperForConditionalGeneration.from_pretrained(
BASE_MODEL, torch_dtype=torch.bfloat16, attn_implementation="sdpa",
)
model = PeftModel.from_pretrained(base_model, ADAPTER)
model = model.to("cuda").eval()
# Transcribe (audio_array: 16kHz float32 numpy array)
input_features = processor(
audio_array, sampling_rate=16000, return_tensors="pt"
).input_features.to("cuda", dtype=torch.bfloat16)
with torch.inference_mode():
predicted_ids = model.generate(
input_features, language="hi", task="transcribe"
)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
For production deployment, merge the adapter and convert to CTranslate2:
# Merge LoRA → convert → evaluate
python convert_and_eval.py --lora-dir outputs/whisper-large-v3-turbo-hindi-lora --quant int8 --gpu 0
from faster_whisper import WhisperModel
model = WhisperModel("path/to/ct2-model", device="cuda", compute_type="int8")
segments, info = model.transcribe("audio.wav", language="hi", beam_size=1)
print(" ".join(seg.text.strip() for seg in segments))
Full pipeline code (data prep → training → deployment): github.com/ipritamdash/whisper-hindi-lora
| Parameter | Value |
|---|---|
| Rank (r) | 32 |
| Alpha | 64 (2x rank) |
| Dropout | 0.05 |
| Target Modules | q_proj, k_proj, v_proj, out_proj, fc1, fc2 |
| Trainable Parameters | 27,852,800 / 836,730,880 (3.33%) |
| Bias | none |
Architecture choice follows LoRA-Whisper (arXiv:2406.06619): encoder+decoder targeting on all linear layers outperforms decoder-only or q/v-only configurations.
| Parameter | Value |
|---|---|
| Base Model | openai/whisper-large-v3-turbo (809M params) |
| Dataset | google/fleurs hi_in |
| Train / Val / Test | 2,120 / 239 / 418 samples |
| Epochs | 3 |
| Learning Rate | 1e-4 (linear decay) |
| Warmup Steps | 50 |
| Batch Size | 4 (x4 gradient accumulation = effective 16) |
| Optimizer | AdamW (weight_decay=0.01) |
| Precision | BFloat16 |
| Gradient Checkpointing | Enabled |
| Hardware | NVIDIA A10G (23GB VRAM) |
| Training Time | 45 minutes |
| Seed | 42 |
Google FLEURS Hindi (hi_in):
Hindi ASR evaluation is sensitive to text normalization. Whisper's default normalizer strips diacritics and simplifies conjunct consonants, which can inflate apparent accuracy but loses semantic precision.
WER numbers above use Whisper-default normalization for comparability with other HuggingFace models. For production Hindi ASR, consider evaluation with IndicNLP normalizer.
language="hi" explicitly.whisper-large-v3-turbo carry through.@misc{dash2026whisper_hindi_lora,
author = {Pritam Dash},
title = {Whisper Large-v3-Turbo Hindi LoRA Fine-tune},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/Tachyeon/whisper-large-v3-turbo-hindi-lora}
}