whisper-small-si-cpu

Hi, I’m Seniru Epasinghe 👋

I’m a AI undergraduate and an AI enthusiast, working on machine learning projects and open-source contributions.
I enjoy exploring AI pipelines, natural language processing, and building tools that make development easier.

---

🌐 Connect with me

        

Sinscribe

This model is a fine-tuned version of openai/whisper-small on the Sinhala CSV + FLACs dataset. It achieves the following results on the evaluation set:

• Loss: 0.0829
• Wer: 29.1387

Intended uses & limitations

Can be used for Sinhala speech to text conversions. Make sure to input noise low audio to the model, to get the best outcome.

Training and evaluation data

Trained on the custom dataset - seniruk/sinscribe-sinhala-stt

Trained on above final dataset with 2 epochs on a device with below spec for 41:00:59 hours

• 16GB RAM
• NVIDIA GeForce RTX 4060 Ti 16GB
• 12th Gen Intel(R) Core(TM) i5-12400F (2.50 GHz) - 64-bit

Training results

Inferencing

import torch
import soundfile
import torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor

device = "cpu"
torchaudio.set_audio_backend("soundfile") 

model = WhisperForConditionalGeneration.from_pretrained("seniruk/whisper-small-si-cpu").to(device)
processor = WhisperProcessor.from_pretrained("seniruk/whisper-small-si-cpu")

def transcribe(audio_path):
    try:
        if audio_path is None:
            return "No audio received. Please record something."

        waveform, sample_rate = torchaudio.load(audio_path)

        if waveform.shape[0] > 1:
            waveform = waveform.mean(dim=0, keepdim=True)

        if sample_rate != 16000:
            resampler = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=16000)
            waveform = resampler(waveform)
            sample_rate = 16000

        waveform = waveform.squeeze().numpy()

        inputs = processor(waveform, sampling_rate=sample_rate, return_tensors="pt").input_features.to("cpu")
        predicted_ids = model.generate(inputs)
        transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

        return transcription

    except Exception as e:
        return f"Error during transcription: {e}"

print(transcribe('audio.wav'))

Gradio UI

import torch
import soundfile
import torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor
import gradio as gr

device = "cpu"
torchaudio.set_audio_backend("soundfile")

model = WhisperForConditionalGeneration.from_pretrained("seniruk/whisper-small-si-cpu").to(device)
processor = WhisperProcessor.from_pretrained("seniruk/whisper-small-si-cpu")

MAX_DURATION_SECONDS = 30  # Limit: 30 seconds of audio

def transcribe(audio_path):
    try:
        if audio_path is None:
            return "No audio received. Please record or upload a file."

        # Load audio
        waveform, sample_rate = torchaudio.load(audio_path)

        # Convert to mono
        if waveform.shape[0] > 1:
            waveform = waveform.mean(dim=0, keepdim=True)

        # Duration check
        duration = waveform.shape[1] / sample_rate
        if duration > MAX_DURATION_SECONDS:
            return f"Audio too long ({duration:.1f}s). Please use an audio clip shorter than {MAX_DURATION_SECONDS}s."

        # Resample if necessary
        if sample_rate != 16000:
            resampler = torchaudio.transforms.Resample(orig_freq=sample_rate, new_freq=16000)
            waveform = resampler(waveform)
            sample_rate = 16000

        waveform = waveform.squeeze().numpy()

        # Process through Whisper
        inputs = processor(
            waveform,
            sampling_rate=sample_rate,
            return_tensors="pt"
        ).input_features.to(device)

        with torch.no_grad():
            predicted_ids = model.generate(inputs)
            transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

        return transcription

    except Exception as e:
        return f"Error during transcription: {e}"


iface = gr.Interface(
    fn=transcribe,
    inputs=gr.Audio(sources=["microphone", "upload"], type="filepath", label="Record or Upload Audio"),
    outputs=gr.Textbox(label="Transcription"),
    title="Whisper Small Sinhala (CPU)",
    description=(
        "🎙️ Sinhala speech-to-text using the fine-tuned Whisper Small model (Sinscribe).\n"
        "You can record or upload audio up to 30 seconds long."
    ),
)

iface.launch()

GPU runnable version of above model is below

• seniruk/whisper-small-si

Framework versions

• Transformers 4.48.0
• Pytorch 2.5.1+cu121
• Datasets 3.6.0
• Tokenizers 0.21.4