whisper-medium-ch-hz

Whisper For Huzhou Dialect

Fine-tuned version of Openai's Whisper speech to text model for the Huzhou Chinese dialect.

Motivation

The inspiration for this project came from my grandmother, who primarily speaks the Huzhou dialect. While looking for existing ASR tools, I found that no ASR systems (including Whisper) were unable to accurately transcribe the Huzhou dialect of Chinese. Since the publicly available models focus on Mandarin and Cantonese, they completely fail on many regional Chinese dialects that are still widely spoken.

So, I decided to build an ASR model that could understand the Huzhou dialect.

Dataset

The model was fine-tuned on approximately 15 hours of raw online video content in the Huzhou dialect. The data was collected from publicly available sources and processed as follows:

• Audio was extracted and segmented from long-form video (primarily from movies and news)
• Subtitle-based alignment (using an OCR model) was used to generate initial labels
• Clips were manually filtered and cleaned to reduce noise and misalignment

Performance

Baseline (Before Fine-Tuning)

The base Whisper model (openai/whisper-medium) performs extremely poorly on the Huzhou dialect. On the evaluation data, it achieved an average Word Error Rate (WER) of ~120.

A WER of 120 indicates that, on average, transcribing a reference sentence of 100 characters would require 120 insertions, deletions, or substitutions to match the ground truth. This essentially means that the model totally fails to understand the Huzhou dialect.

After Fine-Tuning

After fine-tuning on the Huzhou dialect dataset:

• Average WER on testing dataset: 26.77
• This corresponds to roughly 70–80% character accuracy in predicted transcriptions

While not perfect, this represents a substantial improvement and demonstrates that Whisper can be adapted to underrepresented Chinese dialects with relatively modest amounts of data.

Demo

There is a demo notebook that allows you to test the model on sample audio or record your own speech.

Note: Inference may take some time, as the demo currently runs on CPU by default.

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/BillWang7/whisper-medium-ch-hz
cd whisper-medium-ch-hz
conda env create -f environment.yml
conda activate whisper

Run demo.ipynb using the newly created "whisper" conda environment. Either record yourself speaking or upload an audio sample. There are audio samples available in the "test_data.zip".