whisper-small-santali-coded

whisper-small-santali-coded

A full fine-tune of openai/whisper-small for automatic speech recognition on Santali, transcribed into Sanlish — a Latin-script romanization scheme for Santali speech.

Model description

Unlike partial-freeze / adapter-style approaches, every parameter of the base Whisper-small checkpoint (encoder, decoder, and output head) was fine-tuned. No language tag was pinned during tokenization or generation (generation_config.language = None), so the model was not steered toward Bengali or any other Whisper-supported language — it learned Sanlish surface forms purely from the fine-tuning data.

• Base model: openai/whisper-small (~244M params)
• Fine-tuning strategy: full fine-tune, no frozen layers
• Task: speech transcription (audio → Sanlish text)
• Language tag: none pinned (language-agnostic decoding)

Training data

• Source: custom Santali audio + Sanlish transliteration dataset
• Splits: train / validation / test (santali_train.csv, santali_val.csv, santali_test.csv)
• Format: audio_path (16kHz audio) → sanlish (target transliteration text)

Training procedure

Evaluation results

Per-epoch validation metrics (normalized WER/CER — lowercased, punctuation-stripped):

Note: validation loss bottoms out around epoch 3 and rises afterward while WER/CER keep improving slightly — a sign of overfitting on the loss objective that hasn't yet hurt transcription accuracy. Epoch 14 had the lowest validation WER of the run; epoch 17 (the last checkpoint pushed before the training runtime disconnected) is marginally behind it and is what's currently on main. Earlier epoch checkpoints are available in this repo's commit history.

Test set: 28.82% WER, 4.89% CER — evaluated on the held-out test split (193 examples), using the epoch-17 checkpoint currently on main.

Limitations

• Training did not complete its full 20-epoch schedule (interrupted at epoch 17), and early stopping had not yet triggered.
• Fine-tuned on a relatively small custom dataset for a low-resource language variant; may not generalize to speakers, dialects, or recording conditions outside the training distribution.
• No language tag is pinned, so downstream users should explicitly pass task="transcribe" at inference and not rely on Whisper's automatic language detection for this checkpoint.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor

model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-coded")
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-coded", task="transcribe")

model.generation_config.language = None
model.generation_config.task = "transcribe"

# input_features = processor(audio_array, sampling_rate=16000, return_tensors="pt").input_features
# predicted_ids = model.generate(input_features)
# transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)