typhoon-whisper-medium

typhoon-whisper-medium

Thai Whisper-medium with contextual biasing support via a decoder LoRA. Merged base weights + adapter in one repo.

Full fine-tune of openai/whisper-medium on ~10,846 hours of Thai speech (see the Typhoon ASR Real-time technical report).

Results

Character Error Rate (%, lower is better), spaces macro-CER protocol.

Contextual biasing (thai-contextasr-bench, 2,062 utts)

bias@0 = gold entities only (an oracle upper bound). bias@10 = gold plus 10 distractors, the realistic condition. dis@10 = distractors only, measuring over-biasing.

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "typhoon-ai/typhoon-whisper-medium"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(
    model_id, dtype=torch.bfloat16).to("cuda").eval()

feats = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
feats = feats.to("cuda", torch.bfloat16)
ids = model.generate(feats, language="th", task="transcribe", max_new_tokens=440)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Contextual biasing

Load the bundled LoRA adapter (contextual biasing lives in the adapter; the base model alone is a plain Thai ASR), then pass a bias list through Whisper's initial prompt (<|startofprev|>) in the training carrier format.

⚠️ Use this exact carrier format. The model was trained only with the prompt "คำศัพท์ที่เกี่ยวข้อง: " + " ".join(entries) (space-joined entries after the carrier phrase). Any other phrasing, separator, or format was never seen in training and will degrade or disable the biasing effect.

from peft import PeftModel
model = PeftModel.from_pretrained(model, model_id, subfolder="adapter")

prompt = "คำศัพท์ที่เกี่ยวข้อง: " + " ".join(entries)
ids = processor.get_prompt_ids(prompt, return_tensors="pt").to(model.device)
out = model.generate(features, language="th", task="transcribe", prompt_ids=ids)

Architectural limit: Whisper's prompt is capped at 224 tokens inside a 448-token decoder window. Lists beyond ~15–20 entries truncate and degrade sharply — this is a property of the architecture, not of this fine-tune. Keep lists short; retrieve the top-k entries per utterance if you have a large vocabulary.

Limitations

• Domain: >95 % of the training corpus is GigaSpeech2-th (Thai YouTube). In-domain accuracy (gs2_bench) is far stronger than out-of-domain robustness (TVSpeech).
• Benchmark overlap: 3.0 % of gs2_bench utterances share boilerplate transcripts (sponsor taglines, programme intros) with training data. No audio clip is reused — id overlap is 0 — and this affects every system trained on GigaSpeech2-th equally, but read the in-domain number with it in mind. TVSpeech is 100 % clean.
• Thai only. Fine-tuned exclusively on Thai; other languages are not preserved.
• 30 s window. Standard Whisper limit; longer audio needs chunking.

Acknowledgements

We thank Kunat Pipatanakul and Potsawee Manakul (wayu-ai) for their valuable feedback.