whisper-vi-telephony-lora-encoder

Whisper-small VI — Telephony-robust LoRA (encoder-only)

A PEFT LoRA adapter that makes the Vietnamese noise-augmented Whisper-small (banhchungtuongot/whisper-small-vi-en-noise-aug, "Exp C") robust to telephony degradation (8 kHz band-limit + µ-law + babble + reverb) without forgetting clean speech.

LoRA is applied to the encoder linear layers only (decoder fully frozen) and trained multi-condition with an encoder-consistency loss (keep the adapted encoder ≈ the frozen base encoder on clean inputs). The adapter is selected on a held-out validation set disjoint from the test set (no test leakage).

Results (FLEURS vi test, beam-5)

Telephony −27% vs base; clean preserved (≈ base). Dominates full fine-tune on both axes at ~47× fewer trainable parameters. Telephony test set = FLEURS vi + telephony_sim (8k/µ-law/babble/reverb, ESC-50 held-out cats 40–49, SNR 0–15).

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

base = WhisperForConditionalGeneration.from_pretrained("banhchungtuongot/whisper-small-vi-en-noise-aug")
model = PeftModel.from_pretrained(base, "banhchungtuongot/whisper-vi-telephony-lora-encoder")
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
processor.tokenizer.set_prefix_tokens(language="vi", task="transcribe")
# decode with num_beams=5 for the reported numbers

Deploy tip: keep one clean base + hot-swap this 21 MB adapter for the telephony / call-center domain.

Training code & full analysis: https://github.com/NguyenDucAnforwork/asr-h100-noise-robust (training/finetune_whisper_tele_lora_v2.py, reports/telephony_lora_finetune.md).

Framework versions

• PEFT 0.13.2