whisper-small-torgo-dysarthria-lora

Whisper Small — LoRA Adapter for Dysarthric Speech

Parameter-efficient fine-tuning of OpenAI's Whisper Small for clinical speech recognition, targeting dysarthria (TORGO) with speaker-independent evaluation.

Standard ASR models fail catastrophically on clinical speech populations. This adapter reduces Whisper's error rate on dysarthric speech by ~50% using only 1.7M trainable parameters (0.73% of the model).

Quick Start

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
import librosa

processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(model, "dbarbera/whisper-small-torgo-dysarthria-lora")

audio, sr = librosa.load("audio.wav", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
generated_ids = model.generate(**inputs)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

Results

Overall

By Speech Status

By Speaker

Training Configuration

Why LoRA on the decoder only?

The encoder's acoustic representations generalise well across speakers. The decoder's language model is where Whisper fails on clinical speech — it "corrects" valid dysarthric productions toward standard words. Adapting only the decoder teaches the model to listen to what was actually said.

Evaluation methodology

• Text normalisation: Lowercase, strip punctuation, Unicode NFKD (applied to both reference and hypothesis before WER)
• SemScore: Cosine similarity of sentence embeddings (all-MiniLM-L6-v2) — captures semantic correctness even when surface form differs
• Speaker-independent splits: No speaker appears in both train and test sets

Dataset

TORGO — 15 speakers (8 dysarthric, 7 control), ~13.5 hours total, 16,552 utterances.

Available on HuggingFace: abnerh/TORGO-database

Training Hardware

• GPU: NVIDIA B200 (178.4 GB VRAM)
• CUDA: 12.8
• GPU peak memory: 5636 MB
• CPU: AMD EPYC 9555 64-Core Processor
• RAM: 2267.4 GB total, 1983.4 GB available
• Training time: 1286.6s (21.4 min)
• Throughput: 9.71 steps/s
• Platform: Linux-6.8.0-101-generic-x86_64-with-glibc2.39
• PyTorch: 2.8.0+cu128
• Transformers: 5.3.0

Limitations

• Trained on English dysarthric speech only (TORGO corpus, Canadian English)
• Severe dysarthria (e.g., speaker M04) still has high WER — the adapter helps but cannot fully compensate
• Not validated on other clinical populations (aphasia, hearing impairment, etc.)
• LoRA adapter adds minimal inference overhead but requires PEFT library

Citation

@misc{barbera2026whisper-clinical,
  author = {Barbera, David},
  title = {Whisper-Clinical: Parameter-Efficient Fine-Tuning for Dysarthric Speech Recognition},
  year = {2026},
  url = {https://github.com/DavidBarbera/whisper-clinical-speech},
  note = {LoRA adapter for OpenAI Whisper Small trained on TORGO}
}

Author

David Barbera — PhD Cognitive Neuroscience (UCL). Specialising in speech recognition for clinical populations. Built a CE-marked Class II medical device for aphasia rehabilitation.

Website · Google Scholar · GitHub · HuggingFace

---

Last updated: 2026-03-23