whisper-small-hi-kn-en

Whisper Small Fine-Tuned for Indian Speech

This model is based on Whisper Small and was fine-tuned on a mixture of Indian speech datasets.

The goal was to improve transcription quality across Indian languages while maintaining reasonable performance on English speech, noisy audio, and faster speakers.

Training Data

The model was trained using approximately 9,000 audio samples, with 3,000 samples selected from each dataset:

• Shrutilipi
• IndicVoices-R
• FLEURS English

FLEURS was included to help retain English transcription performance during multilingual fine-tuning.

Data Selection

Audio samples were filtered to keep recordings between:

• 5 seconds
• 15 seconds

For IndicVoices-R, additional sampling was used to create a more useful and varied training set.

Noisy Speech

To avoid training on extremely low-quality recordings, samples were selected from the top 20% by signal-to-noise ratio, among the available noisy samples.

This keeps some background-noise variation while avoiding audio that is too difficult or unreliable.

Fast Speech

Samples were also selected from the top 60% by speaking rate.

This was done to expose the model to faster speech and reduce the chance of performance degrading when speakers talk quickly.

Training Objective

The model was fine-tuned for automatic speech recognition using the standard Whisper sequence-to-sequence transcription objective.

The main goals were:

• Better transcription of Indian-language speech
• Improved handling of noisy recordings
• Better performance on faster speakers
• Retaining English transcription ability
• Avoiding very short or excessively long training samples

Intended Use

This model can be used for:

• Automatic speech recognition
• Transcription of Indian-language audio
• Transcription of English audio
• Testing ASR performance on noisy or fast speech
• Further fine-tuning on domain-specific speech data

Limitations

The model was trained on a relatively small subset of each dataset.

Performance may vary depending on:

• Language
• Accent
• Background noise
• Recording quality
• Speaker speed
• Domain-specific vocabulary
• Code-mixed speech

The model may also produce incorrect words, omit parts of speech, or hallucinate text in very noisy or unclear audio.

It should not be used in high-stakes applications without additional evaluation.

Evaluation

The model should be evaluated using metrics such as:

• Word Error Rate
• Character Error Rate

For a fair comparison, references and predictions should use the same text-normalization process.

Evaluation results can be added here:

Base Model

This model was fine-tuned from:

openai/whisper-small

Usage

import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="YOUR_USERNAME/YOUR_MODEL_NAME",
    device=0 if torch.cuda.is_available() else -1,
    dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)

result = pipe(
    "audio.wav",
    generate_kwargs={
        "task": "transcribe",
    },
)

print(result["text"])

For multilingual audio, specifying the language may improve results:

result = pipe(
    "audio.wav",
    generate_kwargs={
        "language": "hindi",
        "task": "transcribe",
    },
)

Acknowledgements

This model was trained using data from:

• Shrutilipi
• IndicVoices-R
• FLEURS
• OpenAI Whisper

Please refer to the original datasets and base model repositories for their respective licenses and usage conditions.