vietnamese-whisper-asr

Vietnamese Speech Recognition with Whisper

Fine-tuned openai/whisper-small (244M parameters) for Vietnamese automatic speech recognition (ASR).

Model Details

• Architecture: Whisper-small (encoder-decoder transformer)
• Parameters: 244M
• Base Model: openai/whisper-small
• Task: Automatic Speech Recognition (Vietnamese)
• Framework: PyTorch + HuggingFace Transformers

Training

• Dataset: Mozilla Common Voice (Vietnamese subset)
• Audio Preprocessing: 16kHz resampling, Gaussian noise injection, VAD-based silence trimming via Librosa
• Augmentation: SpecAugment (frequency + time masking)
• Optimizer: AdamW with linear warmup scheduling
• Gradient Accumulation: 4 steps
• Evaluation Metrics: Word Error Rate (WER), Character Error Rate (CER)

Audio Pipeline

Raw Audio → 16kHz Resampling → Silence Trimming (VAD)
         → Noise Augmentation → Log-Mel Spectrogram
         → SpecAugment → Whisper Encoder-Decoder

Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

model = WhisperForConditionalGeneration.from_pretrained("sanvo/vietnamese-whisper-asr")
processor = WhisperProcessor.from_pretrained("openai/whisper-small")

audio, sr = librosa.load("vietnamese_audio.wav", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
predicted_ids = model.generate(inputs.input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Features

• Vietnamese-optimized speech recognition
• Microphone + file upload Gradio demo
• WER/CER evaluation pipeline
• Audio preprocessing with Librosa (resampling, noise augmentation, silence trimming)

Links

• GitHub: svn05/vietnamese-whisper-asr