whisper-medium-en-vi-accent

Whisper Medium (English) - Fine-Tuned for Vietnamese-Accented English

This model is a fine-tuned, standalone merged version of openai/whisper-medium.en specifically adapted for English speech spoken with a Vietnamese accent.

It was trained on native Vietnamese speakers from the L2-ARCTIC speech corpus using Parameter-Efficient Fine-Tuning (LoRA rank=32, alpha=64), with the learned adapters permanently merged into the base model weights.

---

Benchmark & Performance Evaluation

Evaluated against the baseline whisper-medium.en and whisper-large-v3-turbo on clean held-out read speech (test split):

• Relative WER reduction on held-out Vietnamese-accented English: ~39.4% improvement over zero-shot whisper-medium.en.

---

Training Data & Configuration

• Base Model: openai/whisper-medium.en (769M parameters)
• Dataset: L2-ARCTIC (Vietnamese subset)
• Training Utterances: 4,072 utterances (~4.3h)
• Speakers in Corpus: HQTV (Male), PNV (Female), THV (Female), TLV (Male)
• Acoustic Input: 80-channel log-Mel spectrogram, 16 kHz mono audio
• Training Method: LoRA ($r=32, \alpha=64$, targeting q_proj, k_proj, v_proj, out_proj, fc1, fc2)
• Precision: FP16 merged weights (compatible with standard WhisperForConditionalGeneration)

---

Quickstart & Usage

This is a standalone model. You can load and use it directly with Hugging Face transformers without needing peft or any extra setup.

1. Using pipeline (Recommended)

from transformers import pipeline

# Initialize the pipeline
transcriber = pipeline(
    "automatic-speech-recognition",
    model="Grenmango/whisper-medium-en-vi-accent",
    chunk_length_s=30,
    device="cuda",  # or "cpu"
)

# Transcribe an audio file (automatically resampled to 16kHz)
result = transcriber("path/to/audio.wav")
print(result["text"])

2. Direct Model & Processor Usage

import torch
import soundfile as sf
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "Grenmango/whisper-medium-en-vi-accent"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

# Load 16kHz audio
audio_data, sample_rate = sf.read("path/to/audio.wav")
if sample_rate != 16000:
    import soxr
    audio_data = soxr.resample(audio_data, sample_rate, 16000)

input_features = processor(audio_data, sampling_rate=16000, return_tensors="pt").input_features.to("cuda", torch.float16)

# Generate transcription
predicted_ids = model.generate(input_features, max_new_tokens=128)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print("Transcription:", transcription)

---

Multi-Accent & Personalized Collection

This model is part of a complete multi-accent & personalized English fine-tuning suite on Hugging Face:

• 🇻🇳 whisper-medium-en-vi-accent (Vietnamese)
• 🇸🇦 whisper-medium-en-arabic-accent (Arabic)
• 🇨🇳 whisper-medium-en-chinese-accent (Chinese)
• 🇮🇳 whisper-medium-en-hindi-accent (Hindi)
• 🇰🇷 whisper-medium-en-korean-accent (Korean)
• 🇪🇸 whisper-medium-en-spanish-accent (Spanish)
• 👤 whisper-medium-en-vi-hqtv-personalized (Personalized - HQTV)