whisper-medium-karakalpak

🎙️ Whisper Medium Fine-Tuned for Karakalpak (Qaraqalpaqsha)

📄 Paper: Karakalpak Speech Recognition with Whisper: An Open 107-Hour Corpus, a Held-Out Benchmark, and Knowledge Distillation into Smaller Models (Zenodo, DOI 10.5281/zenodo.23291730)

This repository contains OpenAI Whisper Medium fine-tuned on the training split of the Karakalpak Speech Corpus (93.7 h of native Karakalpak read speech). The validation split was used only to pick the checkpoint and the test split was scored once.

Smaller, faster versions distilled from this model: whisper-small-karakalpak (242M) and whisper-base-karakalpak (73M).

---

🎯 Project Mission: Preserving the Karakalpak Language

In modern artificial intelligence, low-resource languages face a real risk of digital exclusion. Without public datasets and dedicated models, voice assistants, speech-to-text engines, and educational tools remain unavailable to native speakers.

Our project addresses this divide by:

1. Preserving and revitalizing the Karakalpak language: building an open, large-scale ASR baseline.
2. Promoting AI for Social Good: Showing how youth-led initiatives can solve regional and linguistic technology gaps.

---

👥 Project Leadership & Team (Muhammad al-Khwarizmi Specialized School, Nukus)

This initiative was founded and spearheaded by Atabek Kadirbergenov, who assembled and led a student research team from the Muhammad al-Khwarizmi Specialized School in Nukus:

• Atabek Kadirbergenov — Founder, Project Lead & AI/ML Engineer
  Responsibilities: Conceived and founded the initiative, assembled the research team, pitched school leadership to secure GPU computing resources, designed the end-to-end data pipeline and phonetic text normalization algorithms, led Whisper-Medium fine-tuning, benchmarked evaluations, and managed the open-source release.
  Telegram: @atik_uwu | GitHub

• Sanjar Tleumuratov — Telegram Bot, Systems & Data Moderation Lead
  Responsibilities: Crowdsourcing bot architecture, backend server infrastructure, contributor administration, submission quality moderation (filtering invalid/spam recordings), and audio ingestion pipeline.
  Telegram: @Sanjar030609 | LinkedIn

• Dauitbay Nasiredinov — Media, Outreach & PR Lead
  Responsibilities: Regional public relations, community engagement, promotional video production, editing, and mobilizing 200+ native contributors.
  Telegram: @Dawitbay_Nasiratdinov | LinkedIn

• Shohjahon Isroilov (Tashkent Presidential School) — AI/ML Engineering & Data Validation
  Responsibilities: AI/ML engineering support across model training and evaluation, and validation of the collected speech data.
  Telegram: @nottakene | GitHub

• Rasul Zinatdinov — Data Validation & Proofreading
  Responsibilities: Validation of submitted recordings and proofreading of the dataset transcripts.
  Telegram: @gerrorttx | LinkedIn

---

🙏 Acknowledgements

We express our deepest gratitude to:

• Muhammad al-Khwarizmi Specialized School (Nukus): For GPU computing infrastructure and for actively supporting our initiative across the academic community (Official School Announcement).
• The Karakalpak Community: All 200+ contributors across Karakalpakstan who recorded and donated their voices.
• Kaggle: this model was trained on Kaggle's free GPU tier.

---

📊 Results

Scored on the test split (1,074 utterances, 5.3 h) and the validation split, with the same normalisation for every system: lower-cased, punctuation removed, news-agency byline ÓzA removed from references and hypotheses. 95% confidence intervals from a 1,000-sample bootstrap.

Speed (real-time factor = processing time ÷ audio length, greedy): 0.023 on one NVIDIA T4 GPU (fp16), 1.28 on a 4-core CPU at batch size 1. For CPU use, see the distilled whisper-small-karakalpak (RTF 0.47).

---

🥇 Comparison with other Karakalpak ASR systems (test split, greedy decoding)

This test set is in-domain for our model (same text sources and recording pipeline as its training data) and out-of-domain for every other system, so the gap reflects domain match as well as model quality.

---

⚙️ Training Details

• Data: train split only, 19,255 utterances (93.7 h) after removing clips over 30 s
• Base model: openai/whisper-medium (764M parameters)
• Language token: uz (Karakalpak is not a Whisper language; a kk + Cyrillic ablation made no consistent difference)
• Recipe: 6 epochs, effective batch 32, learning rate 1.5e-5 with 500 warm-up steps and linear decay, weight decay 0.01, 8-bit AdamW, fp16, gradient checkpointing, SpecAugment
• Checkpoint selection: lowest validation WER, evaluated every 500 steps (best: step 2,500, epoch 4.2)
• Hardware: 2× NVIDIA T4 (Kaggle), 9.8 h

---

🚀 Quick Start

import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="atikuwu/whisper-medium-karakalpak",
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device="cuda:0" if torch.cuda.is_available() else "cpu",
)

# Always pass language="uz" (the proxy token used in training). Keep timestamps off.
result = pipe("sample_audio.ogg", generate_kwargs={"task": "transcribe", "language": "uz"})
print(result["text"])

Add "num_beams": 5 to generate_kwargs for beam search (9.94% test WER instead of 10.66%).

Beam search with the 4-gram language model (9.47% test WER)

lm/kaa-4gram.arpa is a word 4-gram (modified Kneser–Ney, KenLM) trained on the normalised training transcripts only. code/kaa_asr.py contains ArpaLM (a dependency-free ARPA reader) and LMFusion (a LogitsProcessor for shallow fusion):

import sys; sys.path.insert(0, "code")          # after downloading this repo
from transformers import LogitsProcessorList
from kaa_asr import ArpaLM, LMFusion

lm = ArpaLM("lm/kaa-4gram.arpa")
fusion = LogitsProcessorList([LMFusion(lm, pipe.tokenizer, alpha=0.1, beta=1.0)])
result = pipe("sample_audio.ogg", generate_kwargs={"task": "transcribe", "language": "uz",
                                                     "num_beams": 5, "logits_processor": fusion})

alpha=0.1, beta=1.0 were tuned on validation for this model. In our evaluation, outputs with a gzip compression ratio above 2.4 (repetition loops, 0–1 per 1,074 utterances) were re-decoded greedily without the LM.

---

📂 Repository contents

• model.safetensors, configs, tokenizer: the fine-tuned model
• lm/kaa-4gram.arpa: 4-gram language model (train transcripts only)
• code/: training, distillation and evaluation code used for these results (Kaggle kernels)
• eval/: scripts used for the comparison with other systems

---

📜 Citation

@misc{kadirbergenov2026karakalpakasr,
  title={Karakalpak Speech Recognition with Whisper: An Open 107-Hour Corpus, a Held-Out Benchmark, and Knowledge Distillation into Smaller Models},
  author={Kadirbergenov, Atabek and Tleumuratov, Sanjar and Nasiredinov, Dauitbay and Isroilov, Shohjahon},
  year={2026},
  publisher={Zenodo},
  doi={10.5281/zenodo.23291730},
  url={https://zenodo.org/records/23291730}
}