whisper-large-v3-turbo-lt

whisper-large-v3-turbo-lt 🇱🇹

Fine-tune of openai/whisper-large-v3-turbo for Lithuanian automatic speech recognition, trained on the LIEPA-3 corpus.

Public benchmarks (out-of-domain) — the crown numbers

Measured on standard public test sets the model was not trained on (LIEPA-3 ≠ FLEURS / Common Voice), so these reflect real-world generalisation, not the optimistic in-domain number further below. Whisper-normalised WER (lowercase, punctuation stripped), greedy decoding, language="lt".

Roughly halves (FLEURS) to two-thirds (Common Voice) the base model's error, and beats prior public Lithuanian Whisper fine-tunes (commonly ~20 % on Common Voice).

In-domain results (optimistic)

In-domain held-out set (400 clips excluded from training, n=400) — same corpus as training, so this number is optimistic; trust the public benchmarks above for real-world quality:

A −87 % relative WER reduction. CER (1.56 %) is reported because Lithuanian is highly inflected — a wrong case ending is a full word error but ~1 character error, so WER overstates the damage; CER is the fairer morphological signal.

WER curve over training (in-domain held-out, every 2 000 steps):

step     0    2k    4k    6k    8k   10k   12k   14k   16k   18k   20k
WER%  49.66 18.21 15.21 13.08 10.56 9.69  9.08  8.32  7.91  7.17  6.64

Usage

from transformers import pipeline
asr = pipeline("automatic-speech-recognition", model="svogunas/whisper-large-v3-turbo-lt")
print(asr("audio.wav", generate_kwargs={"language": "lithuanian", "task": "transcribe"}))

Known limitations

Measured 2026-08-07, after a user reported the first one. Please read this before wiring the model into a pipeline — it will save you the debugging session it cost them.

No timestamps at all

This model emits no timestamp tokens. Not inaccurate ones — none. So anything that asks for timing comes back empty:

• vLLM with response_format="verbose_json" → no timestamped segments
• transformers with return_timestamps=True → a single chunk that never closes: (0.0, None). That leading 0.0 is inserted by WhisperTimeStampLogitsProcessor, not predicted by the model, and you will see the warning "Whisper did not predict an ending timestamp"
• subtitle generation, word-level alignment, or search-by-time → will not work directly

Plain transcription is unaffected and works normally.

Measured over 8 concatenated items of ~5 minutes each (40.8 min, 211 known segment boundaries), compared against the base model under an identical harness:

On short clips the picture is the same. Over 150 clips each, the base model returned a complete (start, end) pair on 150/150; this fine-tune managed 5/150 on FLEURS-lt and 22/150 on CommonVoice — sporadic enough to be unusable, and a direct probe shows (0.0, None) on every clip in a 6-clip sample against (0.0, 17.46) from the base on the same 18.4 s audio.

Cause (updated 2026-08-07): a bug in our training code, not a limitation of the data. Our trainer tokenized every target with WhisperTokenizer's default prefix, which is <|startoftranscript|><|lt|><|transcribe|><|notimestamps|>. So all 20 000 steps explicitly supervised "emit no timestamps". The model does exactly what it was taught. generation_config.json is byte-identical to the base model, so there is nothing to fix on your side.

If you fine-tune Whisper yourself, check this: it is a one-line default and it silently costs you the entire timestamp capability. Pass tokenizer.set_prefix_tokens(predict_timestamps=...) per row and train on a mixture; we measured that 500 steps is enough to bring the competence back.

If you need timestamps, use paprika-whisper-lt-v3 instead

kristijonas/paprika-whisper-lt-v3 is a third-generation fine-tune of this model by the kalamo.ai team, retrained on labels that carry timestamps. We verified the fix independently on 2026-08-18 rather than take the card's word for it, by counting timestamp token ids in the generated sequence — decoded text is not trustworthy here, because a tokenizer loaded from a different repo silently drops timestamp tokens, and the transformers ASR pipeline clamps the last chunk's end to the audio length, which makes any model that closes at all look perfectly accurate:

On long audio the difference is not subtle. Same harness as the table further up — 8 items of ~5 minutes (40.8 min, 211 known segment boundaries), chunk_length_s=30:

The items hold ~27 source clips each, so v3's segment count also tracks the truth, while the base model over-segments (up to 51).

So v3 does not merely recover the competence, it places boundaries far more precisely than the base model does. (Our items are clips joined by 0.3 s of silence, which makes boundaries easier to find than natural speech would — but every model above was scored on the identical audio.) Its long-form WER on that set is 18.70 chunked, against 22.95 for this model.

⚠️ One caution from the same runs: v3's native long-form decoding looped catastrophically on 1 of 8 five-minute items (a 5-gram repeated 218 times, 41 % more words than the reference), while chunked decoding never did in any run. A second 8-item build did not loop at all, so the trigger is the audio, not a setting — Whisper's classic long-form failure has not gone away, and you should check your own. Its card prescribes temperature fallback and condition_on_prev_tokens=False, which is sensible insurance; on the build where we could compare, those settings cost ~2 WER points and about 5× the runtime with no loop present to prevent.

⚠️ One packaging caveat if you go there on transformers 4.x: both paprika repos ship only the newer processor_config.json and no preprocessor_config.json, so WhisperProcessor.from_pretrained("kristijonas/paprika-whisper-lt-v3") raises OSError: does not appear to have a file named preprocessor_config.json — including in their own quickstart. Load the processor from this repo instead (identical feature extractor and tokenizer) or move to transformers 5.x.

We have no corrected weights of our own in the pipeline: that fine-tune already exists, it is downstream of this one, and racing it for a few tenths of a WER point would serve nobody.

If you must stay on this model — segment with a VAD, transcribe each segment, and take the timings from the VAD:

# pip install silero-vad
import torch, soundfile as sf
from transformers import pipeline

audio, sr = sf.read("audio.wav")  # 16 kHz mono
vad, utils = torch.hub.load("snakers4/silero-vad", "silero_vad")
(get_speech_timestamps, *_) = utils
segments = get_speech_timestamps(torch.tensor(audio, dtype=torch.float32), vad, sampling_rate=sr)

asr = pipeline("automatic-speech-recognition", model="svogunas/whisper-large-v3-turbo-lt")
for s in segments:
    text = asr({"raw": audio[s["start"]:s["end"]], "sampling_rate": sr},
               generate_kwargs={"language": "lithuanian", "task": "transcribe"})["text"]
    print(f'[{s["start"]/sr:.2f}s - {s["end"]/sr:.2f}s] {text}')

You get segment-level timings from the VAD and Lithuanian accuracy from the model. For word-level timings you would need a Lithuanian forced aligner; whisperX does not ship one for lt.

No punctuation or casing

Output is lowercase and unpunctuated, for the same reason — the LIEPA-3 labels have neither, across all 7.2 M lines, by corpus policy.

Pair it with a restoration model. We measured 1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase, which covers lt, on 2026-08-07 — readable WER, meaning no normalisation, punctuation and casing both counted:

Comma F1 is 86.7 on news prose and 81.2 on read literary speech, but it breaks on medical dictation (comma F1 49.7, roughly doubling the comma count). Genuinely spontaneous Lithuanian is untested — no punctuated reference for it exists.

Long-form audio

Prefer chunk_length_s=30 for audio longer than 30 seconds. Native sequential long-form decoding depends on timestamp tokens, which this model does not emit, and measures ~7 points of WER worse than chunked on the 5-minute items above.

⚠️ Chunked decoding has its own failure mode, measured and reported by the paprika authors: where the overlap between two windows cannot be aligned, transformers discards the span, silently — they observed 30 and 52 dropped words at seams on clean audio, with nothing in the output marking the gap. It also narrates non-speech, producing confident Lithuanian on digital silence. Gate silence before decoding. For this model the trade is still worth it, because native decoding needs the timestamps it does not have; if that trade bothers you, the v3 fine-tune above decodes natively.

Training

• Base: openai/whisper-large-v3-turbo
• Data: LIEPA-3 (read + spontaneous), ~7.23 M clips, on-the-fly feature extraction over a shuffled manifest (no full-epoch — a fixed 20 000-step budget over broad diversity)
• Hyperparameters: 20 000 steps · effective batch 16 (bs 8 × grad-accum 2) · LR 1e-5 linear (warmup 500) · bf16 · gradient checkpointing
• Hardware: single NVIDIA DGX Spark (GB10)

License & attribution

Trained on LIEPA-3, distributed under CC BY 4.0 by VDU / CLARIN-LT. You must attribute VDU / CLARIN-LT when using this model. Model weights: CC BY 4.0.