whisper-large-v2-formosan-lang-tokens

Model Card for whisper-large-v2-formosan-lang-tokens

This model is a fine-tuned version of ILRDF/whisper-large-v2-formosan-all (itself based on openai/whisper-large-v2) for automatic speech recognition across 42 Formosan indigenous language dialects of Taiwan.

Unlike the base model - which conditions every dialect on a single placeholder Whisper language token (id, Indonesian) - this model adds one real Whisper-style language token per dialect (e.g. <|ami-x-frng|>), the same mechanism Whisper itself uses for its original ~99 languages, just extended to 42 dialects it was never pretrained on. Each dialect's own added token replaces the usual <|lang|> slot in the decoder prefix, so the model is told exactly which dialect it's transcribing instead of having to infer it (or default to one shared representation for all of them).

Supported dialects

42 dialects across 15 language families of Taiwan's Formosan indigenous languages. Each has its own added language token, replacing the <|lang|> slot in the decoder prefix <|startoftranscript|><|lang|><|transcribe|><|notimestamps|>.

<details> <summary>Full list of 42 dialects and their tokens (click to expand)</summary> </details>

Results

Compared against the base model, ILRDF/whisper-large-v2-formosan-all, on the eval split of every config of formospeech/klokah and formospeech/ithuan_formosan (45 dialect/dataset pairs total). Numbers below are normalized WER/CER (lowercased, light punctuation stripped - see "Text normalization" in the details below for exactly what that means).

Macro-average (mean of each dialect's own metric) across all 45, and separately per dataset:

This model beats the baseline's normalized WER/CER on 44/45 and 43/45 dialects respectively (the handful of losses are within ~0.1-0.5 points); on raw, non-normalized WER/CER it wins 45/45.

Noise robustness (MUSAN noise mixed in at the target SNR before transcription, same 45 pairs, macro-averaged):

<details> <summary>Full per-dialect results & methodology (click to expand)</summary>

Every number above and in the per-dialect table below is broken down per dataset, per config (dialect), and per split; the pooled figures are macro-averages computed from those per-dialect numbers, not a single blended metric. The model-index metadata in this README's YAML frontmatter carries the same breakdown machine-readably: one results entry per (dataset, config, split) triple, 45 in total, each with its own dataset.type/dataset.name/dataset.config/dataset.split and wer/cer (raw and normalized) metrics - this is what powers the auto-rendered results box on this model's Hub page. Both datasets only ship train/eval splits, and eval is the one used everywhere here - there is no separate held-out test split.

Note on the model-index format used here vs. Hugging Face's Evaluation Results docs: that page documents a newer, separate mechanism (.eval_results/*.yaml files, tied to a dataset repo that's registered as a Hub Benchmark with its own eval.yaml) built for community-submitted, semi-verified leaderboard-style results. Neither formospeech/klokah nor formospeech/ithuan_formosan is registered as a Benchmark, so that mechanism doesn't apply to this model card. The model-index: block above is the older, more widely-supported convention (the same one used to render the standard "Results" table on model pages across the Hub, including the sibling formospeech/whisper-large-v2-taiwanese-hakka-v1 model card) and is what's actually used here.

Text normalization

Both models' predictions and references are lowercased, stripped of ., ,, !, ?, ; and have runs of whitespace collapsed to a single space before computing the normalized metrics - deliberately a small, fixed whitelist rather than stripping every Unicode punctuation/symbol character (e.g. transformers' BasicTextNormalizer), because two characters that look like punctuation are actually meaningful orthography in this data and would otherwise get silently corrupted:

• : (colon) marks vowel length in some dialects (Amis, Saisiyat) - e.g. bae:iw, sapi:ihin. A corpus-wide scan found it directly between two letters (i.e. part of a word, not separating sentences) 77-95% of the time in the dialects that use it.
• U+2303 (⌃) is a meaningful orthographic marker that also occurs in the data.

Neither is stripped. Every other punctuation character actually appearing in the training corpora was confirmed (via the same scan) to never occur mid-word, so stripping just those five is safe.

Noise robustness methodology

MUSAN noise-subset clips (bilguun/musan-noise, 930 clips) are additively mixed in at a target SNR before transcription, same 45 dialect/dataset pairs as above. Raw (non-normalized) WER/CER for the same three conditions:

All four metrics (WER/CER, raw and normalized) degrade monotonically as SNR decreases, as expected. The Rukai (dru-*) dialects are consistently the most noise-sensitive of the 42, with dru-x-opnh and dru-x-kgdv losing 15-20 points of absolute WER at 10dB SNR - worth keeping in mind for any downstream use in noisy conditions. This is one condition among many possible robustness probes (there's no single agreed-upon standard for which MUSAN subset(s) or SNR range to use for ASR robustness evaluation specifically - MUSAN itself was originally designed for training-time augmentation recipes, not a fixed eval protocol). Results here should be read as "robustness to point-source background noise at moderate SNR," not a full robustness certification.

Per-dialect breakdown

</details>

Usage

Access and Authentication

This model is hosted as a gated Hugging Face repository. Before using it:

1. Visit the model page and request access.
2. Log in with the same Hugging Face account that has been granted access.
3. Authenticate your local environment with a Hugging Face access token.

A read token is sufficient for inference.

pip install -U huggingface_hub
hf auth login

Alternatively, you can provide the token through the HF_TOKEN environment variable:

export HF_TOKEN=hf_xxx

Do not hard-code your Hugging Face token in scripts, notebooks, or public repositories.

If you see an error such as Cannot access gated repo, make sure that:

• your Hugging Face account has been granted access to this model;
• hf auth whoami shows the expected account;
• HF_HUB_DISABLE_IMPLICIT_TOKEN is not set.

Run model

Because each dialect uses its own added language token instead of one of Whisper's built-in languages, the usual pipeline(..., generate_kwargs={"language": ...}) shortcut does not apply here - the convenience language=/task= arguments only recognize Whisper's original ~99 languages. Instead, force the dialect's token directly via decoder_input_ids:

import json
import torch
from huggingface_hub import hf_hub_download
from transformers import WhisperForConditionalGeneration, WhisperProcessor

model_id = "formospeech/whisper-large-v2-formosan-lang-tokens"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32

processor = WhisperProcessor.from_pretrained(model_id, language="id", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch_dtype).to(device)

# lang_code -> added token id, e.g. "ami-x-frng" -> 51865 (see "Supported dialects" above)
lang_code_to_token_id = json.load(open(hf_hub_download(model_id, "lang_code_to_token_id.json")))

def transcribe(audio_array, sampling_rate, lang_code):
    decoder_start_id, _, transcribe_id, notimestamps_id = processor.tokenizer.prefix_tokens
    dialect_token_id = lang_code_to_token_id[lang_code]
    # Passing decoder_input_ids to Whisper's generate() is taken verbatim (it does NOT
    # auto-prepend decoder_start_token_id the way the base GenerationMixin does), so all
    # four prefix tokens must be given explicitly here.
    decoder_input_ids = torch.tensor(
        [[decoder_start_id, dialect_token_id, transcribe_id, notimestamps_id]], device=device
    )
    input_features = processor.feature_extractor(
        audio_array, sampling_rate=sampling_rate, return_tensors="pt"
    ).input_features.to(device, dtype=torch_dtype)
    with torch.no_grad():
        generated_ids = model.generate(input_features, decoder_input_ids=decoder_input_ids, max_length=225)
    return processor.tokenizer.decode(generated_ids[0], skip_special_tokens=True)

# example: audio_array is a float32 numpy array at 16kHz
# transcribe(audio_array, 16000, "ami-x-frng")

Training process

The training of the model was performed with the following hyperparameters:

• Hardware: 4x NVIDIA RTX A5000
• Per-device batch size: 2
• Gradient accumulation steps: 64
• Effective batch size: 512
• Total training steps: 1159 (1 epoch)
• Learning rate: 1e-4
• Warmup ratio: 0.1
• Precision: bf16
• Optimizer: adamw_torch
• LR scheduler type: linear
• Initialization: base model's weights (ILRDF/whisper-large-v2-formosan-all), with the 42 new dialect token embeddings each initialized from that model's own id (Indonesian) token embedding - a linguistically related Austronesian language, per Whisper maintainer guidance on adding new languages

The learning rate was chosen empirically via a short LR probe (3e-5 / 1e-4 / 3e-4 candidates, ~100 steps each) before committing to a full run - 1e-4 gave the best eval WER among the three.

Training data

Every config (dialect) of every dataset's train split was loaded and concatenated:

• formospeech/ntu_formosan_corpus
• formospeech/ilrdf_dicts
• formospeech/klokah
• formospeech/ithuan_formosan

Notes

• This release contains inference files only. Optimizer states and trainer checkpoints are intentionally excluded.
• lang_code_to_token_id.json (included in this repo) is required to look up the correct token id per dialect - see the usage example above.