svara-whisper-small-hi

svara-whisper-small-hi

Whisper-small fine-tuned on 60,000 Hindi utterances from AI4Bharat's IndicVoices.

On Voice Arena's Monsoon Hindi test set, scored with the Open ASR Leaderboard's own harness, it gets 43.6% OIWER against 241.6% for the base model. On FLEURS Hindi it scores WER 0.434 against 0.815. Most of the gap is one failure: whisper-small falls into repetition loops on spontaneous Hindi, and this model almost never does.

It is a standalone Whisper model, with the LoRA adapter merged in, so it loads like any other Whisper checkpoint.

Versions. This is the second release. The first, trained on 20,000 utterances, scored 53.7% OIWER and WER 0.476. It is still available as revision v1-20k:

pipeline("automatic-speech-recognition", model="cruqAI/svara-whisper-small-hi", revision="v1-20k")

The CTranslate2 int8 export at cruqAI/svara-whisper-small-hi-ct2 was made from that first version and has not been redone for this one.

Results: Monsoon hi (spontaneous speech)

VoiceArena/Monsoon_hi_test, 753 utterances, 1.33 hours, scored by running open_asr_leaderboard's transformers/run_eval_ml.py unmodified on one RTX 3090. The metric is OIWER, which counts a word correct if it matches any spelling in that slot's reference lattice. This is the leaderboard's own benchmark and code, run by us; it is not an official leaderboard submission.

The two fine-tuned rows were scored in the same run on the same card. The base model row comes from an earlier run on a different 3090 (where v1 scored the same 53.7% at RTFx 45.1), so compare its error rate, not its speed.

An error rate over 100% means the model emitted more wrong words than the reference has words. That is what happens with the base model: it produces a repetition loop on 264 of the 753 clips, one of them 148 words of "तो". Those loops alone add 22,231 words to a 13,006-word reference, which is 171 of the 241 points. v1 loops on 4 clips and this model on 1.

Results: FLEURS hi (read speech)

FLEURS Hindi test split, 418 utterances, 1.34 hours of read speech. Greedy decoding with the decoder's own 448-token limit, fp16 on one GPU. All rows were scored by the same code with the same text normalisation (danda to full stop, punctuation and case removed, numerals expanded), so the comparison is fair but the absolute numbers are not the official Whisper ones.

On 33 of the 418 clips the base model repeats a word or phrase until it runs out of tokens (one output is the word "तो" 90 times); v1 did that on none. So most of the distance from the base model is the loops, and the step from v1 to this version is ordinary recognition getting better.

Two caveats. FLEURS is read speech and the training data is mostly conversational and spontaneous, so this is an out-of-domain test; expect different numbers on conversational audio. And fp16 inference is not bit-stable across GPUs: the same model scored within 0.005 of itself on different cards, so differences that small are noise.

Training data

The first 60,000 rows, in corpus order, of the Hindi training split of ai4bharat/IndicVoices (CC-BY-4.0). About 50,300 of them were used for training, after the filters (among them, clips outside 0.5 to 30 seconds and one transcript longer than Whisper's 448-token decoder limit) and a held-out development slice of 3,288. IndicVoices is crowd-collected: read, extempore and conversational speech from many speakers and districts, with the noise and code-mixing that implies.

The FLEURS and Monsoon test sets were never used for training or model selection.

Training

• Base: openai/whisper-small (244M parameters)
• Method: LoRA on the attention query and value projections, rank 32, alpha 64, dropout 0.05 (3.5M trainable parameters), then merged into the base weights
• 3 epochs, 9,435 optimizer steps, effective batch size 16 (8 x 2 accumulation)
• Learning rate 1e-5 with 500 warmup steps, fp16, no label smoothing
• About 2 hours 15 minutes on one RTX 3090 rented through Nosana, resumed once from a checkpoint
• Final development-set loss 0.412
• Training text was normalised with the project's training profile, which keeps punctuation; scoring used the stricter profile described above

The code, data pipeline and the exact run configuration are in github.com/cruq-ai/svara.

Use

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="cruqAI/svara-whisper-small-hi",
    generate_kwargs={"language": "hindi", "task": "transcribe"},
)
print(asr("clip.wav")["text"])

Audio is resampled to 16 kHz by the pipeline. Clips longer than 30 seconds need chunk_length_s=30 or return_timestamps=True, as with any Whisper model.

Limitations

• Hindi only. The fine-tune was single-language and other Whisper languages were not evaluated after it.
• Trained on 60,000 utterances, still a fraction of IndicVoices Hindi.
• The scoring normaliser expands numerals to words, so a transcript with digits is penalised against a reference with words and vice versa.
• The Monsoon numbers come from the leaderboard's harness run on our own hardware, not from an official leaderboard submission.
• Both test sets are about 1.3 hours. Differences of a point or two are not meaningful at that size.

License and attribution

The model weights are released under Apache-2.0, the same licence as the base model. The training data is CC-BY-4.0: AI4Bharat, IndicVoices. Evaluation used FLEURS (CC-BY-4.0), Google, and Voice Arena's Monsoon hi test set.