whisper-fleurs-zh-asr

Whisper-Small Fine-tuned for Chinese ASR on FLEURS

This repository contains a Chinese automatic speech recognition (ASR) model based on OpenAI Whisper-small, fine-tuned on the FLEURS Chinese (cmn_hans_cn) dataset.

The project started from a zero-shot Whisper baseline and then evolved into a full fine-tuning and optimization pipeline, including:

• baseline construction and error analysis
• supervised fine-tuning (V1)
• data cleaning, speed perturbation, scheduler tuning, label smoothing, and staged encoder freezing (V2)

---

Model Overview

• Base model: openai/whisper-small
• Task: Chinese ASR
• Dataset: google/fleurs (cmn_hans_cn)
• Framework: PyTorch + Hugging Face Transformers
• Evaluation metric: Character Error Rate (CER)

---

Key Results

Baseline

Zero-shot Whisper-small baseline on FLEURS Chinese validation set:

• Validation CER (normalized): 23.64%

V1 Fine-tuning

After implementing a supervised fine-tuning pipeline:

• Validation CER (normalized): 11.31%
• Test CER (normalized): 12.35%

V2 Optimization

After introducing sample cleaning, speed perturbation, cosine scheduler, label smoothing, and staged encoder freezing:

• Validation CER (normalized): 10.29%
• Test CER (normalized): 11.36%

---

What Was Improved in V2

Compared with V1, V2 introduced several optimization strategies:

• conservative sample cleaning
• speed perturbation augmentation
• cosine learning rate scheduler
• label smoothing
• staged encoder freezing
• stronger decoding settings during final evaluation

The training set was expanded from 3242 cleaned samples to 4362 training examples after augmentation.

---

Dataset

This model was trained and evaluated on:

• Dataset: FLEURS
• Configuration: cmn_hans_cn

Data split used in the project:

• train: 3246 raw samples
• validation: 409 samples
• test: 945 samples

---

Training Pipeline

The project was implemented as a full experimental pipeline rather than a simple tutorial reproduction. The workflow included:

1. building a zero-shot baseline
2. defining normalized CER evaluation
3. exporting per-sample predictions for error analysis
4. implementing supervised fine-tuning
5. saving the best checkpoint based on validation CER
6. performing V2 optimization with augmentation and training strategy tuning

---

Inference

Example usage:

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch
import torchaudio

model_id = "elonmush/YOUR_MODEL_REPO"

processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id)

waveform, sr = torchaudio.load("your_audio.wav")
if waveform.ndim == 2:
    waveform = waveform.mean(dim=0)

inputs = processor(
    waveform.numpy(),
    sampling_rate=sr,
    return_tensors="pt"
)

with torch.inference_mode():
    pred_ids = model.generate(
        inputs.input_features,
        language="chinese",
        task="transcribe",
        max_new_tokens=128,
    )

text = processor.batch_decode(pred_ids, skip_special_tokens=True)[0]
print(text)