yecs-asr-ctc-lid

YECS ASR — CTC with Inline Language-Tag Injection (yo/en)

A Yoruba–English code-switching ASR model that jointly transcribes speech and labels the language of every word inline, e.g.:

<en> first aid </en> <yo> ni wọ́n ṣáré fún kí ó tó kú </yo>

It is omniASR_CTC_300M (Meta Omnilingual ASR, ~330M params) fine-tuned on the YECS Yoruba–English code-switching corpus, with four atomic tag tokens (<yo> </yo> <en> </en>) added to the SentencePiece vocabulary (9812 → 9816) and the CTC head resized accordingly. Language-boundary prediction is learned as an auxiliary task at zero extra inference cost — one model does transcription + per-word LID.

Why tags? (controlled result)

Two omniASR_CTC_300M models were fine-tuned identically (5 epochs, same 80,013 utterances / 95.6 h, same recipe) and scored with identical normalization on the same full held-out 9,949-utterance YECS test set (tags stripped from both refs and hyps before WER):

Injecting language tags does not hurt transcription — it slightly improves WER/CER (~3.3% relative) while delivering 98% per-word language ID for free.

Normalization

WER/CER use NFC + lowercase + Unicode P/S punctuation stripping. "Tone-aware" keeps Yoruba tone diacritics; "tone-insensitive" strips acute/grave/macron but keeps the phonemic under-dot (ọ/ẹ/ṣ). Tags are removed from both reference and hypothesis before scoring the transcription; language tags are scored separately by aligning ref/hyp word sequences and comparing each aligned word's language.

Usage

This is a fairseq2 checkpoint (not 🤗 Transformers). Load with the omnilingual-asr stack using the included asset cards, then run the inference pipeline. The checkpoint (model.pt), extended tokenizer (omniASR_tokenizer_v1.model) and cards are in this repo.

from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
import torch
pipe = ASRInferencePipeline(model_card="omniASR_CTC_300M_lid_ft",
                            device=torch.device("cuda"), dtype=torch.bfloat16)
print(pipe.transcribe(["utt.wav"], batch_size=8))
# -> ['<en> ... </en> <yo> ... </yo>']

Training

• Base: omniASR_CTC_300M; 5 epochs (3,125 steps), lr 1e-5, bf16, grad-accum 4, 1×H200.
• Targets serialized into contiguous same-language spans; tags are atomic SentencePiece user-defined symbols (survive decoding, unlike control tokens).
• W&B: afroscale_ai_cmu_africa/yecs-lid (run ctc-tag-inject).

Evaluation note

Scored on the full held-out YECS test split (9,949 utts); test audio is the canonical Mozilla/MDC release. The test reference labels are held privately by LyngualLabs (the public test_metadata.csv ships with targets stripped), so these metrics are from an internal evaluation and are not independently reproducible without those labels.

License / data

Model weights derive from Meta Omnilingual ASR. YECS corpus terms apply to the training data. Intended for research on Yoruba–English code-switching ASR and LID.