whisper-large-v3-turbo-rinpoche

whisper-large-v3-turbo-rinpoche

Tibetan speech-to-text for Garchen Rinpoche's speech, fine-tuned from openai/whisper-large-v3-turbo. Encoder-decoder, 32 encoder / 4 decoder layers, 51,866-token multilingual BPE vocabulary. generation_config.json already sets language="bo" and task="transcribe".

• Task: automatic speech recognition
• Language: Tibetan (bo)
• Input: 16 kHz mono audio, padded to whisper's 30 s window
• Weights: fp16, single model.safetensors (1.6 GB)
• ⚠️ Recommended segment length: under ~9.5 s — see Limitations

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor

REPO = "GarchenArchive/whisper-large-v3-turbo-rinpoche"
processor = WhisperProcessor.from_pretrained(REPO)
model = WhisperForConditionalGeneration.from_pretrained(
    REPO, dtype=torch.float16
).to("cuda").eval()

def transcribe(waveform, sampling_rate=16000):
    feats = processor(waveform, sampling_rate=sampling_rate, return_tensors="pt")
    with torch.no_grad():
        ids = model.generate(feats.input_features.to("cuda", torch.float16))
    return processor.tokenizer.decode(
        ids[0], skip_special_tokens=True, clean_up_tokenization_spaces=False
    )

Pass clean_up_tokenization_spaces=False when decoding: the default cleanup is designed for WordPiece and is destructive for BPE output.

Run inference in fp16. The fp32 master OOMs a 12 GB card during beam search, in the KV-cache reorder.

Evaluation

Measured on the test split of GarchenArchive/stt-rinpoche-202609 (1,394 utterances): the 554 utterances of the earlier stt-rinpoche-202605 test split (202605 subset) plus 840 round-2 rows. All numbers below come from the production STT code path. Audio is cut into ≤ 9 s chunks at quiet points, as production does, decoded greedily in fp16, and the chunks are joined with a space.

CER via jiwer. WER and SER (syllable error rate) via tibetan_wer 2.0, micro averages. tibetan-wer 2.0 changed syllable segmentation, so SER and WER are not comparable with earlier versions of this card. CER is unaffected.

Paired bootstrap, this release minus previous: SER −0.0068 (95% CI −0.0127 to −0.0011) and CER −0.0015 (−0.0061 to +0.0030). The SER gain is significant; the CER gain is within noise. The gain comes from the shorter round-2 clips. The 202605 subset is slightly worse, but not significantly.

The previous release remains available as revision 2cbf9032, tagged v202609. Pin it with revision="v202609".

Fused with a CTC model

The production pipeline reconciles this model's output with GarchenArchive/wav2vec2-xls-r-300m-rinpoche using an LLM (gemini-3-flash-preview, with neighbouring segments as context). This beats either model alone. The two models fail differently, which is what makes the combination work.

The fused result is essentially unchanged (ΔCER −0.0004, 95% CI −0.0039 to +0.0033). Repeated runs of the LLM step alone vary by about 0.001 CER.

Limitations

Output length ceiling. Whisper's decoder is capped at 448 positions and Tibetan tokenises at roughly 43–47 tokens per second of speech in this vocabulary, so audio beyond ~9.5 s cannot fit. The failure is silent: the model emits a well-formed EOS and stops early. You cannot detect it by looking for a missing EOS, because whisper's pad and EOS token ids are identical. Detect it by comparing output length against clip duration.

This dominates the model's error profile, and the effect is visible directly in the deletion counts by subset:

Same model, same decoding — deletion-dominance is a property of clip length, not of the model. Segment your audio short and it largely disappears.

Beam search returns a degenerate n-best. With num_beams=5, num_return_sequences=5, the returned hypotheses have been observed to be identical with identical sequences_scores. Beam search itself works — rank 0 differs from greedy — but the n-best list is not usable for reranking.

Domain. Fine-tuned on a single speaker in a specific register. Prior in-house experiments found that further fine-tuning on general Tibetan corpora (Kham dialect, NICTTIB1) degrades performance substantially.

Training

This release continues the previous one (2cbf9032) with a fresh LoRA adapter, merged back into the weights.

Data: GarchenArchive/stt-rinpoche-202609 train (14,426 human-transcribed utterances, 21.9 h), plus 2,528 pseudo-labelled clips (3.8 h, all 7 s or shorter).

The pseudo-labels are LLM-fused transcripts of untranscribed archive recordings from the 2026-10-19 batch run. They were kept only where the reference-free CER estimate was 0.10 or lower. That filter selects clips the ensemble is most confident about. Unfiltered pseudo-labels made this model worse. With the filter, the model beats a continuation trained on human data alone, over two seeds: pooled CER −0.0027, 95% CI −0.0054 to −0.0001. Released: the seed-43 run. Details are in RnD/ASR/Tibetan/Experiments/PseudoLabel202610 in garchen-archive/ArchiveAI-RnD.

The adapters were merged into the base weights and saved as fp16, so this repo is a drop-in replacement and needs no peft at inference.

LoRA was used rather than a full fine-tune because full AdamW on 809 M parameters needs about 12.9 GB before activations. The training card has 11.6 GB usable.

Training targets longer than the 448-token budget were split at Tibetan clause boundaries (།) rather than truncated. Transcriber annotations that were never spoken (❓, parenthesised alternate readings, ༼ ༽) were stripped from the training text.