whisper-large-v3-sk

whisper-large-v3-sk

Slovak fine-tune of openai/whisper-large-v3 by the KInIT team. Full parameter fine-tuning on a curated Slovak speech corpus, with noise augmentation for robustness to real-world recording conditions. Listed in SLAIH, a catalog of Slovak NLP resources.

Model Details

Intended Use

This model is intended for Slovak automatic speech recognition across a range of domains and recording conditions.

Out-of-scope: Non-Slovak audio, real-time streaming without appropriate chunking, safety-critical transcription without human review.

Evaluation

Word Error Rate (WER) and Character Error Rate (CER), lower is better. Measured on two Slovak eval sets:

• CV24 - Common Voice 24.0 test split (5,239 samples, public)
• Internal - held-out KInIT set (9,317 samples, stratified by domain and speaker gender, one third clean and two thirds noise-augmented; not public)

Before vs. after fine-tuning

Slovak fine-tuning reduces WER by 61% on Common Voice and 65% on the internal eval set, at the cost of multilingual performance (see Limitations).

Choosing a model size

Comparison of all six Whisper sizes before and after Slovak fine-tuning. WER is measured on our internal benchmark dataset (see above):

The full KInIT ASR collection includes all six Whisper sizes plus the NVIDIA NeMo fine-tunes (Canary, Parakeet).

Version History

• v2 (current): Added noise augmentation during fine-tuning (75% of training batches) for improved robustness to real-world recording conditions.
• v1: Original release, full fine-tuning without noise augmentation.

Loading a specific release

Every release is tagged: v2.0 for the current release, v1.0 for the original one. main always points at the current release, so the Usage example below loads v2.

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

repo_id = "kinit/whisper-large-v3-sk"
revision = "v2.0"  # "v1.0" for the original release

model = AutoModelForSpeechSeq2Seq.from_pretrained(repo_id, revision=revision)
processor = AutoProcessor.from_pretrained(repo_id, revision=revision)

Pin revision in production code if you need reproducible results across future releases.

Training Data

Fine-tuned on an internal curated Slovak speech corpus compiled at KInIT. The corpus combines public datasets with internal KInIT recordings. Recordings containing personal data were anonymised prior to use. Samples were quality-filtered using a CER-based threshold validated against multiple ASR models. During training, 75% of batches were additionally augmented on the fly with synthetic background noise for robustness to real-world recording conditions (see Training Procedure below).

Training Procedure

Training was performed on the Devana HPC cluster.

Usage

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

model_id = "kinit/whisper-large-v3-sk"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    dtype=dtype,
    use_safetensors=True,
).to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    dtype=dtype,
    device=device,
)

result = pipe("audio.wav", generate_kwargs={"language": "slovak"})
print(result["text"])

License

Apache 2.0

Limitations

• Catastrophic forgetting: Fine-tuning exclusively on Slovak data significantly degrades performance on other languages. Use the base openai/whisper-large-v3 if multilingual transcription is required.
• Performance may degrade on strongly accented, dialectal, or domain-specific speech not represented in the training data.
• Maximum reliable single-segment length is 30 seconds without chunking.

Acknowledgements

Public datasets used in training: SloPalSpeech, Mozilla Common Voice, FLEURS, and the TEDxSK and JumpSK Lecture Speech Corpus (KEMT NLP).

(Part of the) Research results was obtained using the computational resources procured in the national project National competence centre for high performance computing (project code: 311070AKF2) funded by European Regional Development Fund, EU Structural Funds Informatization of society, Operational Program Integrated Infrastructure.