whisper-base-mla-cv11

Whisper-Base-MLA (11 languages) — MLA-converted, 62.5% smaller decode KV-cache

The on-device-tier sibling of the WhisperMLA family: openai/whisper-base (74M) with its decoder self-attention converted MHA→MLA (per Whisper-MLA, arXiv:2603.00563), recovery-fine-tuned on 11 languages of the CC0 Whispered corpus (32k clips/lang).

from transformers import AutoModelForSpeechSeq2Seq
model = AutoModelForSpeechSeq2Seq.from_pretrained("burakaydinofficial/whisper-base-mla-cv11", trust_remote_code=True)  # transformers==4.46.x

Honest sizing note (read this first)

Conversion cost grows as the student shrinks — measured across the family: small ≈ +0.4 median WER → base ≈ +1.0 → tiny ≈ +1.9. At the base tier you pay ≈ +1.0 WER (median) for the 62.5% cache cut. If quality is the priority, prefer the small variant; this tier is for memory-constrained deployments where the cache cut matters most.

Results (CommonVoice-17 test, n=1500/lang; WER/CER %; cost = paired vs an identically-trained unconverted control)

Absolute quality is base-tier-typical (much lower than small — that is the base model, not MLA). Encoder frozen both arms; 15,000 steps; warmup+cosine; bf16 (weights released as fp16).

Matched control now published — verify the conversion cost yourself: burakaydinofficial/whisper-base-cv11 (trained identically, minus the MHA→MLA conversion). Evaluate both with scripts/validate.py.

Limitations

• Costs are recovery-mix-specific: do NOT compare a language's conversion cost across the cv11 and 24-language lines — different recovery sets move each language's low-rank fit (French in the small model: +0.32 cv11 vs +1.22 24-lang). Not irreproducibility.
• What the 62.5% is (cache scope): it is the decode self-attention KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream total decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
• Requires trust_remote_code=True and transformers==4.46.x (custom MLA attention modules); not loadable in whisper.cpp / faster-whisper / CTranslate2.
• Language coverage is the 11 recovery languages — unseen scripts degrade.
• Georgian (ka) reported as the labeled model-class floor; not usable at this tier.
• Read-speech domain, greedy-decode evals — beam-5 adds ~1-2 WER on both arms without changing the conversion cost (measured on the small tier).

Acoustic conditions of the evaluation

Evaluated on crowdsourced consumer-microphone recordings with real environmental noise — traffic, room reverb, variable devices — CommonVoice's native conditions, not studio audio. The numbers above already include that heterogeneity. Not yet benchmarked: far-field, telephony (8 kHz), overlapping speech; an SNR-ladder robustness section will be added when measured.

FLEURS (out-of-domain) — this model, WER% (CER% for th/zh)

ABSOLUTE FLEURS numbers on a different read-speech corpus. Do NOT compare these per-language to the CommonVoice table above — the corpora differ in transcript normalization/orthography and speaker pools (e.g. Welsh reads easier on FLEURS, Georgian worse), so cross-corpus swings are corpus artifacts, not model behavior. The clean out-of-domain claim is the PAIRED MLA-vs-control cost (verified to be small on the small tier). Per-language absolute: en 16.1 · de 25.8 · es 15.3 · fr 30.4 · ru 30.5 · tr 36.0 · cy 82.0 · ar 55.4 · th 34.7 · zh-CN 23.5 · ka 100.2. (Full JSON: the release repo's eval artifacts.)