finetuned-whisper-small-nepali-ct2

Fine-tuned Whisper Small Nepali

This model is a fine-tuned version of openai/whisper-small for Nepali automatic speech recognition.

The model is intended for Nepali speech-to-text transcription and is part of a larger Nepali speech processing pipeline that can later support:

Nepali speech → Nepali text → punctuation restoration → machine translation → English text/speech

Model Details

• Base model: openai/whisper-small
• Task: Automatic Speech Recognition
• Language: Nepali
• Language code: ne
• Training type: Full fine-tuning
• Framework: Hugging Face Transformers
• License: Apache 2.0

Training Data

The model was fine-tuned using a combination of publicly available Nepali speech datasets:

1. Mozilla Common Voice 26.0

   • Locale: ne-NP
   • Used for Nepali speech recognition fine-tuning

2. Google FLEURS

   • Config: ne_np
   • Used for additional Nepali speech-text examples

The datasets were normalized into a shared schema:

audio
sentence
source

Audio was resampled to 16 kHz. Text was Unicode-normalized and lightly cleaned before Whisper feature extraction.

Transformers Usage

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="sparshrestha/finetuned-whisper-small-nepali",
    generate_kwargs={
        "language": "nepali",
        "task": "transcribe",
    },
)

text = pipe("audio.wav")["text"]
print(text)

GPU Usage

import torch
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1

pipe = pipeline(
    "automatic-speech-recognition",
    model="sparshrestha/finetuned-whisper-small-nepali",
    device=device,
    generate_kwargs={
        "language": "nepali",
        "task": "transcribe",
    },
)

text = pipe("audio.wav")["text"]
print(text)

Gradio Demo

import torch
import gradio as gr
from transformers import pipeline

device = 0 if torch.cuda.is_available() else -1

pipe = pipeline(
    "automatic-speech-recognition",
    model="sparshrestha/finetuned-whisper-small-nepali",
    device=device,
    generate_kwargs={
        "language": "nepali",
        "task": "transcribe",
    },
)

def transcribe(audio):
    if audio is None:
        return ""
    return pipe(audio)["text"]

iface = gr.Interface(
    fn=transcribe,
    inputs=gr.Audio(
        sources=["microphone", "upload"],
        type="filepath",
    ),
    outputs="text",
    title="Whisper Small Nepali",
    description="Demo for Nepali speech recognition using a fine-tuned Whisper small model.",
)

iface.launch(share=True)

CTranslate2 / faster-whisper Deployment

For faster inference, this model can be converted to CTranslate2 format.

Install

pip install faster-whisper ctranslate2

Convert

ct2-transformers-converter \
  --model sparshrestha/finetuned-whisper-small-nepali \
  --output_dir finetuned-whisper-small-nepali-ct2 \
  --copy_files tokenizer.json preprocessor_config.json \
  --quantization float16

Run with faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel(
    "finetuned-whisper-small-nepali-ct2",
    device="cuda",
    compute_type="float16",
)

segments, info = model.transcribe(
    "audio.wav",
    language="ne",
    task="transcribe",
    beam_size=5,
)

text = " ".join(segment.text.strip() for segment in segments)
print(text)

Lower-memory GPU inference

from faster_whisper import WhisperModel

model = WhisperModel(
    "finetuned-whisper-small-nepali-ct2",
    device="cuda",
    compute_type="int8_float16",
)

segments, info = model.transcribe(
    "audio.wav",
    language="ne",
    task="transcribe",
    beam_size=5,
)

text = " ".join(segment.text.strip() for segment in segments)
print(text)