whisper-small-mla-cv11

Whisper-Small-MLA (11 languages) — MLA-converted Whisper with a 62.5% smaller decode KV-cache

The first released MLA-converted Whisper checkpoints. The decoder self-attention of openai/whisper-small is converted from MHA to MLA (multi-head latent attention, the DeepSeek-style KV-compression — here via SVD-based conversion per Whisper-MLA, arXiv:2603.00563), then recovery-fine-tuned on 11 languages of the CC0 Whispered corpus. You get a Whisper whose per-token decoder cache is 62.5% smaller at near-identical accuracy — without training anything yourself.

Quickstart

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "burakaydinofficial/whisper-small-mla-cv11", trust_remote_code=True)          # transformers==4.46.x
processor = AutoProcessor.from_pretrained("burakaydinofficial/whisper-small-mla-cv11")

inputs = processor(audio_16khz, sampling_rate=16000, return_tensors="pt")
ids = model.generate(inputs.input_features,
                     language="de", task="transcribe", max_new_tokens=128)
print(processor.batch_decode(ids, skip_special_tokens=True))

What you gain / what it costs (all measured)

• Decode KV-cache: −62.5% per token (measured from live cache tensors, exact).
• Decode speed: −2% to −18% (batch 64 → batch 1; the win is memory/concurrency, not latency — decode remains far faster than real-time either way).
• Accuracy: conversion cost ≤ +1.0 WER on 10 of 11 languages vs an identically-trained unconverted control (paired per-speaker bootstrap, n=1500/language) — not statistically significant on 5 of them (fr, cy, ar, th; zh on CER). The out-of-domain check (FLEURS) on the study models confirms the pattern is not a CommonVoice artifact.

Results (CommonVoice-17 test, n=1500/lang, greedy, Whisper text normalization + Arabic folding)

Better than stock on 9 of 11 languages, at parity on de, −1.0 on en. The "cost" column is the honest MLA price: this model vs a control trained identically without the conversion.

Size-scaling note: conversion cost grows as the student shrinks (small ≈+0.4 median → base ≈+1.0 → tiny ≈+1.9) — smaller models have less redundancy to absorb the compression. This small variant is the sweet spot of the released family.

Matched control now published — verify the conversion cost yourself: burakaydinofficial/whisper-small-cv11 (trained identically, minus the MHA→MLA conversion). Evaluate both with scripts/validate.py.

Limitations (read before deploying)

• Costs are recovery-mix-specific: do NOT compare a language's conversion cost across the cv11 and 24-language lines — different recovery sets move each language's low-rank fit (French in the small model: +0.32 cv11 vs +1.22 24-lang). Not irreproducibility.

• What the 62.5% is (cache scope): it is the decode self-attention KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream total decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.

• Requires trust_remote_code=True and transformers==4.46.x (custom MLA attention modules). Not loadable in whisper.cpp / faster-whisper / CTranslate2.

• Language coverage is the 11 recovery languages. Fine-tuning erodes Whisper's other ~88 languages — unseen scripts degrade most (measured). Use stock Whisper for languages far from this set.

• Georgian (ka) is included for transparency but not usable at the whisper-small tier.

• Evaluated with greedy decoding; beam-5 adds ~1-2 WER on both arms without changing the conversion cost (measured on the small tier); incremental KV-cache decode validated (matches cache-free).

• Research-grade checkpoint: conversion validated near-lossless at init (full-rank SVD reproduces baseline logits) and the packaged fp16 model reproduces the trained model to within fp16 precision (validated <1e-4 max logit diff in fp32; the release is an fp32→fp16 downcast, not bit-exact), but this is not a hardened production artifact.

Provenance

• Base: openai/whisper-small (Apache-2.0). Conversion code adapted from Whisper-MLA / MHA2MLA (Apache-2.0), with fixes.
• Recovery data: Whispered (CC0) — 32k clips × 11 languages (~350 h), CommonVoice-17-derived with quality gates, random-sampled.
• Training: 15,000 steps, bf16, warmup+cosine LR, gradient clipping, encoder frozen (both arms — the multilingual encoder is deliberately preserved), best checkpoint selected on the dev split. An identically-trained unconverted control was used for all cost claims (paired bootstrap CIs).
• Weights: fp16 safetensors. 8/4-bit loading works via bitsandbytes through transformers.

Acoustic conditions of the evaluation

Evaluated on crowdsourced consumer-microphone recordings with real environmental noise — traffic, room reverb, variable devices — CommonVoice's native conditions, not studio audio. The numbers above already include that heterogeneity. Not yet benchmarked: far-field, telephony (8 kHz), overlapping speech; see the measured SNR section below.

SNR-ladder robustness (measured)

Paired conversion cost at 20 / 10 / 0 dB additive pink noise (en, ar, zh-CN; n=1500/lang; clip-aligned corruption): en +0.8 / +0.4 / +0.9 WER (borderline-significant at the same magnitude as clean), ar and zh not significant at ANY noise level including 0 dB. The compression cost does not grow with noise — this model degrades as gracefully as the unconverted control.

FLEURS (out-of-domain) — this model, WER% (CER% for th/zh)

ABSOLUTE FLEURS numbers on a different read-speech corpus. Do NOT compare these per-language to the CommonVoice table above — the corpora differ in transcript normalization/orthography and speaker pools (e.g. Welsh reads easier on FLEURS, Georgian worse), so cross-corpus swings are corpus artifacts, not model behavior. The clean out-of-domain claim is the PAIRED MLA-vs-control cost (verified to be small on the small tier). Per-language absolute: en 11.0 · de 14.0 · es 9.2 · fr 17.1 · ru 17.6 · tr 21.1 · cy 59.5 · ar 33.1 · th 25.0 · zh-CN 15.4 · ka 97.3. (Full JSON: the release repo's eval artifacts.)