whisper-large-v3-turbo-encoder-pruned

whisper-large-v3-turbo-encoder-pruned

A pruned and distilled variant of openai/whisper-large-v3-turbo with 6 encoder layers removed (layers 5, 6, 7, 9, 11, 12) and recovered via label-free knowledge distillation.

Paper: Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
Code: https://github.com/rasgaard/whisper-encoder-layer-prune

What is this?

The 32-layer Whisper encoder contains redundant middle layers. This model removes the 6 least important layers (identified via leave-one-out ΔWER ranking), then re-aligns the pruned encoder to the original using MSE distillation on unlabelled speech — no transcriptions required.

The result is a drop-in replacement: same architecture family, same tokenizer, same decoder — just a shallower encoder.

Performance

Evaluated on FLEURS test splits. Parentheses show absolute change in percentage points.

Efficiency

Usage

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    "rasgaard/whisper-large-v3-turbo-encoder-pruned",
    torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained(
    "rasgaard/whisper-large-v3-turbo-encoder-pruned"
)

Distillation details

• Teacher: openai/whisper-large-v3-turbo (frozen)
• Objective: MSE on final encoder hidden states
• Data: People's Speech validation split (~18k English utterances, unlabelled)
• Training: 2000 steps, lr=1e-5, AdamW, batch=8, ~26 min on A100
• Best checkpoint: step 1500

@misc{aagaard2026layerslessencoderpruning,
      title={Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery}, 
      author={Rasmus Aagaard and Nicki Skafte Detlefsen},
      year={2026},
      eprint={2609.27980},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2609.27980}, 
}