whisper-small-jiangyin-lora

Whisper Small Jiangyin LoRA Adapter

This is a LoRA adapter for openai/whisper-small fine-tuned on Jiangyin dialect from WuSutra.com.
Wusutra.com is a dialect crowdsourcing website which implements the entire ML workflow — including audio upload, model training, validation, and inference. You can upload your own audios and even trigger the training yourself on wusutra.com. If you have further questions, feel free to message me.

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

# Load base model
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
processor = WhisperProcessor.from_pretrained("openai/whisper-small")

# Load LoRA adapter
model = PeftModel.from_pretrained(model, "jxue/whisper_small_jiangyin_lora")

# Transcribe audio
with torch.no_grad():
    input_features = processor(
        audio_array, sampling_rate=16000, return_tensors="pt"
    ).input_features.to(device)

    predicted_ids = model.generate(
        input_features=input_features,
        forced_decoder_ids=processor.get_decoder_prompt_ids(language="zh", task="transcribe")
    )

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

Training Details

• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.1
• Training samples: 168
• Training epochs: 5
• Learning rate: 3e-4
• CER: 0.00%