whisper-small-lora-ntu-ml2021

Whisper-small LoRA for Mandarin-English Technical Lectures

A rank-32 LoRA adapter that adapts multilingual openai/whisper-small to Mandarin-dominant, code-switched technical speech (Chinese lectures with English machine-learning terms). Built for OpenTypeless, an open-source macOS voice-to-text app.

The adapter is 14 MB, trains in ~30 minutes on a single RTX 2060 (6 GB), and runs comfortably on Apple Silicon. A CTranslate2 conversion for faster-whisper is published at Creaturelove7/faster-whisper-small-ntu-ml2021-ct2.

Results

Evaluated on 11,919 held-out test utterances (the official test split with all dev files removed — the pinned dataset revision ships every dev file inside test, and the evaluation excludes them):

That is a 67% relative CER reduction. On an in-domain dev subset, the adapted 244M model also outperformed the 6x larger whisper-large-v3 served by a cloud API (CER 4.29% vs 9.46%). This is an in-domain result: on out-of-domain audio the general-purpose large model is expected to win.

The output is normalized Simplified Chinese with English technical terms preserved in Latin script, and the normalization survives int8 quantization.

Full evaluation reports (aggregate metrics, pinned revisions, package versions, hashed sample identifiers — no transcripts) are in the training repository.

Usage

from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor

base = "openai/whisper-small"
processor = WhisperProcessor.from_pretrained(base, language="zh", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(base)
model = PeftModel.from_pretrained(model, "Creaturelove7/whisper-small-lora-ntu-ml2021")
model.generation_config.forced_decoder_ids = processor.get_decoder_prompt_ids(
    language="zh", task="transcribe"
)

The decoder prompt is fixed to Chinese transcription, matching training; English technical terms are still emitted in Latin script. Use model.merge_and_unload() for a standalone checkpoint.

Training

• Base model: openai/whisper-small @ 973afd24965f72e36ca33b3055d56a652f456b4d
• Data: ky552/ML2021_ASR_ST @ 1e121cc419e87eed7d4825400baa06f102931944, all 17,779 official train utterances, 1 epoch
• LoRA: rank 32, alpha 64, dropout 0.05 on attention q_proj/v_proj (<1% trainable parameters)
• fp16, effective batch 15, AdamW, lr 1e-4, seed 42
• Labels and inference share the same fixed Chinese-transcription decoder prompt
• Text normalization: Traditional→Simplified conversion, intra-Han space removal

The complete, revision-pinned training and evaluation pipeline (hash-locked dependencies, split-leakage guards, Colab notebook) lives in training/ntu_ml2021.

Limitations

• Domain-adapted to machine-learning lecture speech; expect degraded accuracy on other domains.
• The corpus is predominantly Taiwan-accented Mandarin.
• Optimized for short utterances (a few seconds); the training data caps clips at 60 seconds.
• The fixed Chinese decoder prompt is wrong for English-only dictation.

Data provenance

The training corpus is the public Hugging Face dataset ky552/ML2021_ASR_ST (NTU ML2021 course lecture audio; the dataset card declares MIT). The underlying lecture recordings are the work of the course authors; this repository redistributes no audio, transcripts, or per-sample predictions — only LoRA weight deltas. If you are a rights holder of the source material and want this adapter removed, open an issue and it will be taken down.