yecs-asr-llm-lid

YECS ASR — LLM with Inline Language-Tag Injection (yo/en)

A Yoruba–English code-switching ASR model that jointly transcribes speech and labels the language of every word inline:

<en> customer </en> <yo> sọ pé wọ́n fẹ́ kí </yo> <en> invoice </en> <yo> jẹ́ </yo> <en> separated </en>

It is omniASR_LLM_300M (Meta Omnilingual ASR — wav2vec2 encoder + LLaMA decoder, 1.63B params) fine-tuned on the YECS corpus, with four atomic tag tokens (<yo> </yo> <en> </en>) added to the vocabulary (9812 → 9816) and both the decoder embedding and output head resized. One model does transcription and per-word LID.

Result

Evaluated on the full held-out 9,949-utterance YECS test set with identical normalization (tags stripped from refs and hyps before WER):

Tag injection slightly improves the LLM's WER (16.51 → 16.27, word-prefix 16.18) while adding 99.55% per-word language identification at zero extra inference cost. For reference, the CTC tag-injection variant (LyngualLabs/yecs-asr-ctc-lid) reaches 32.45% WER / 98.04% LID — lighter and faster, lower accuracy.

Usage

fairseq2 checkpoint (not 🤗 Transformers). Load with the omnilingual-asr stack via the included asset cards; pass lang="yor_Latn" for the LLM's language conditioning.

from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
import torch
pipe = ASRInferencePipeline(model_card="omniASR_LLM_300M_lid_ft",
                            device=torch.device("cuda"), dtype=torch.bfloat16)
print(pipe.transcribe(["utt.wav"], lang=["yor_Latn"], batch_size=8))

Training

• Base omniASR_LLM_300M; 5 epochs (7,750 steps), lr 1e-5, bf16, fsdp, grad-accum 4, 1×H200.
• Embedding surgery keeps the special-token row (shifted) and inserts 4 tag rows; head appends 4 rows. Card sets both wav2vec2_asr_config.target_vocab_size and llama_config.vocab_size to 9816.
• Normalization: NFC + lowercase + Unicode P/S strip; tone-aware keeps tone diacritics, tone-insensitive strips acute/grave/macron but keeps the phonemic under-dot.
• W&B: afroscale_ai_cmu_africa/yecs-lid (run llm-tag-inject).
• Code: https://github.com/osinkolu/yecs-asr-benchmark (tag-injection/).

Evaluation note

Scored on the full held-out YECS test split (9,949 utts); test audio is the canonical Mozilla/MDC release. Test reference labels are held privately by LyngualLabs (public test_metadata.csv has targets stripped), so metrics are internal and not independently reproducible. †LLM-plain (LyngualLabs/yecsASR-omni-llm, 16.51) is a separately-trained finetune re-scored here on the same 9,949/pipeline — comparable metric, but not the identical 5-epoch recipe as this tag model. (Its earlier reported 16.15% used a different normalization.)

License / data

Weights derive from Meta Omnilingual ASR; YECS corpus terms apply to the training data. Research use on Yoruba–English code-switching ASR + LID.