whisper-small-jv-id-lora

Whisper Small — Indonesian & Javanese Code-Switching ASR

This model is a fine-tuned version of OpenAI Whisper Small for automatic speech recognition (ASR) on Indonesian and Javanese speech, including code-switched utterances.

Model Details

• Base model: openai/whisper-small
• Task: Automatic Speech Recognition (ASR)
• Languages: Indonesian (id) and Javanese (jv)
• Training method: LoRA (Low-Rank Adaptation)
• Final model: LoRA adapter merged into the base Whisper model
• Model type: Encoder-decoder Transformer
• Framework: Hugging Face Transformers
• Audio sampling rate: 16 kHz

Intended Use

This model is intended for speech-to-text transcription of Indonesian and Javanese audio.

It is particularly useful for:

• Indonesian speech recognition
• Javanese speech recognition
• Indonesian–Javanese code-switching
• Conversational speech transcription
• Multilingual ASR experiments

Training Approach

The model was fine-tuned using LoRA (Low-Rank Adaptation) instead of updating all parameters of the Whisper model.

The training process was:

OpenAI Whisper Small
        ↓
LoRA Fine-Tuning
        ↓
Indonesian + Javanese Speech Data
        ↓
Code-Switched ASR Training
        ↓
LoRA Adapter
        ↓
Adapter Merging
        ↓
Final Merged Whisper Model

The final uploaded model contains the merged weights, so the LoRA adapter does not need to be loaded separately.

Why LoRA?

LoRA allows efficient fine-tuning by training only a small number of additional parameters while keeping most of the original Whisper parameters frozen.

Advantages include:

• Lower GPU memory requirements
• Faster fine-tuning
• Fewer trainable parameters
• Efficient experimentation
• Easy adapter-based training

After training, the LoRA weights were merged with the original Whisper model to create a standalone model.

Model Architecture

The model is based on the Whisper encoder-decoder architecture:

Audio
  ↓
Whisper Feature Extractor
  ↓
Whisper Encoder
  ↓
Whisper Decoder
  ↓
Text Transcription

Usage

Installation

pip install -U transformers torch librosa soundfile

Load the Model

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "manish320/whisper-small-jv-id-lora"

processor = WhisperProcessor.from_pretrained(model_id)

model = WhisperForConditionalGeneration.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

Transcribe Audio

import librosa
import torch

audio, sampling_rate = librosa.load(
    "audio.wav",
    sr=16000
)

inputs = processor(
    audio,
    sampling_rate=16000,
    return_tensors="pt"
)

input_features = inputs.input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(input_features)

transcription = processor.batch_decode(
    predicted_ids,
    skip_special_tokens=True
)[0]

print(transcription)

Example

Input audio:

Saya sedang pergi ke pasar, banjur tuku panganan.

Possible transcription:

Saya sedang pergi ke pasar, banjur tuku panganan.

The model is designed to preserve the language switching present in the speech rather than forcing the complete utterance into a single language.

Language / Decoding

For multilingual and code-switching speech, language forcing can be avoided so that Whisper can generate the transcription naturally.

For example:

predicted_ids = model.generate(
    input_features
)

If you specifically want to force a language, Whisper generation options can also be configured depending on the evaluation setup.

Limitations

The model may have reduced performance on:

• Languages other than Indonesian and Javanese
• Heavy background noise
• Very low-quality recordings
• Strong accents not represented in the training data
• Speakers or domains significantly different from the training dataset
• Very long audio recordings
• Code-switching patterns that are underrepresented in the training data

Performance may also vary depending on microphone quality, speaking style, audio duration, and domain.

Evaluation

Recommended ASR evaluation metrics include:

• Word Error Rate (WER)
• Character Error Rate (CER)

Example:

from jiwer import wer

reference = "reference transcription"
prediction = "model transcription"

score = wer(reference, prediction)

print("WER:", score)

Lower WER indicates better transcription performance.

Base Model

This model is based on:

OpenAI Whisper Small

Whisper is a multilingual automatic speech recognition model trained on a large-scale collection of audio and text data.

Model Files

The repository contains the merged model weights and configuration required to load the model with Hugging Face Transformers.

Typical files include:

config.json
generation_config.json
model.safetensors
preprocessor_config.json
tokenizer_config.json
special_tokens_map.json
vocab.json
merges.txt

License

Please refer to the license of the original Whisper model and the license/terms of the dataset used for fine-tuning.

Acknowledgements

Thanks to:

• OpenAI for the Whisper model
• Hugging Face for the Transformers and PEFT ecosystem
• The creators and contributors of the Indonesian and Javanese speech datasets used for fine-tuning

Citation

If you use this model in your project, please cite the original Whisper work and this model repository.

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Radford et al., 2022

Disclaimer

This model is provided for research and development purposes. Transcription quality may vary depending on the audio, speaker, language, accent, and domain.

---

license: mit language:

• jv, id, multilingual metrics:
• wer

---
license: mit
base_model:
- openai/whisper-small
pipeline_tag: automatic-speech-recognition
tags:
- indonesian
---