yecs-asr-whisper-lid

YECS Whisper-small — Inline Language-Tag Injection (yo/en)

A Yoruba–English code-switching ASR model that jointly transcribes speech and labels the language of every word inline, in a single decoding pass:

<en> first aid </en> <yo> ni wọ́n ṣáré fún kí ó tó kú </yo>

It is openai/whisper-small (244M) fine-tuned on the YECS Yoruba–English code-switching corpus, with four language-tag tokens (<yo> </yo> <en> </en>) added to the tokenizer and the decoder embeddings resized. Language-boundary prediction is learned as an auxiliary task at zero extra inference cost — one model does transcription and per-word language ID.

Results (YECS full held-out test, 9,949 utterances)

Scored against the plain-fine-tuned control with identical normalization. Language tags are stripped from both reference and hypothesis before computing WER/CER, so the transcription comparison is apples-to-apples; the tags are scored separately as LID.

Adding inline tags matches the plain model's WER (within run-to-run noise; CER is actually a touch better) while delivering 99.54% per-word language identification for free. For reference, the reported whisper-small-yoruba baseline is 20.76% WER; this project's plain 5-epoch fine-tune reaches 16.69%.

The same result holds across three architectures

The tag-vs-plain A/B was run identically on three model families:

Inline language-tag injection costs ~nothing on WER (−1.1 to +0.1 absolute) and gives 98–99.5% free per-word LID on CTC, LLM, and Whisper alike.

How it was built

1. Target serialization — each transcript's per-word language_tags are grouped into contiguous same-language spans and wrapped: <yo> … </yo> <en> … </en>.
2. Tokenizer — the four tags are added with add_tokens(..., special_tokens=False) so they are ordinary vocabulary items that survive decode(skip_special_tokens=True) (unlike control tokens, which would be dropped). Embeddings are resized to match.
3. Fine-tune — standard Seq2SeqTrainer, language conditioning yoruba, task transcribe.

Normalization (how WER/CER are computed)

NFC → lowercase → strip Unicode punctuation/symbol categories (P/S). Two variants:

• tone-aware keeps Yoruba tone diacritics (acute/grave/macron), so getting a word's tone wrong counts as an error — the stricter, fuller score.
• tone-insensitive additionally strips the three tonal combining marks, isolating word/segment accuracy. The phonemic under-dot (ọ, ẹ, ṣ) is a distinct letter and is always kept in both variants.

The gap between the two measures how well the model handles Yoruba tone marking.

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch, librosa

repo = "LyngualLabs/yecs-asr-whisper-lid"
proc = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16).to("cuda").eval()

audio, _ = librosa.load("utterance.wav", sr=16000)          # 16 kHz mono
feats = proc.feature_extractor(audio, sampling_rate=16000, return_tensors="pt").input_features
ids = model.generate(feats.to("cuda", torch.bfloat16),
                     language="yoruba", task="transcribe", max_new_tokens=225)
print(proc.tokenizer.batch_decode(ids, skip_special_tokens=True)[0])
# -> '<en> ... </en> <yo> ... </yo>'

Strip the tags for a plain transcript, or parse the spans to recover per-word language.

Training

• Base: openai/whisper-small · 5 epochs · lr 1e-5 · bf16 · effective batch 32 · 1×H200.
• Language conditioning yoruba, task transcribe; forced_decoder_ids=None, suppress_tokens=[]; best checkpoint by dev WER.
• Data: YECS train (80,013 utts / ~95.6 h), 16 kHz mono; dev for checkpoint selection.
• Logged to Weights & Biases (afroscale_ai_cmu_africa/yecs-lid, run whisper-tag).

Intended use & limitations

Research on Yoruba–English code-switching ASR and per-word spoken language identification.

• Trained on YECS read/prompted speech; far-field, noisy, or spontaneous conversational audio may degrade.
• LID is scored on words the model recognizes (aligned ref/hyp words); it is not a standalone LID system for arbitrary audio.
• Two languages only (yo, en); other languages are out of scope.

Related

• Plain baseline: LyngualLabs/yecs-asr-whisper-plain
• Omnilingual variants: yecs-asr-ctc-lid · yecs-asr-llm-lid
• Code & full writeup: https://github.com/osinkolu/yecs-asr-benchmark (tag-injection/)
• Dataset: YECS (LyngualLabs)

Evaluation note

Scored on the full held-out YECS test split (9,949 utts); test audio is the canonical Mozilla/MDC release. Test reference labels are held privately by LyngualLabs (public test_metadata.csv has targets stripped), so metrics are internal and not independently reproducible.

License

Apache-2.0 (inherited from openai/whisper-small). YECS corpus terms apply to the training data.