omniasr-ctc-300m-kk

omniASR-CTC-300M Kazakh (fine-tuned)

A Kazakh-specialised CTC speech-recognition model, fully fine-tuned from omniASR_CTC_300M_v2 (the 300M-parameter CTC member of Meta's Omnilingual ASR family, pretrained on 1600+ languages) on Kazakh speech from ISSAI Kazakh Speech Corpus 2 and the Kazakh split of Google FLEURS.

Unlike the multilingual base model, this checkpoint is specialised for Kazakh: it more than halves Word Error Rate on Kazakh at the cost of degrading on other languages (see Limitations). It is a full fine-tune — the entire encoder and the CTC projection head were trained — not an adapter.

• Base model: omniASR_CTC_300M_v2 (Apache 2.0)
• Tokenizer: omniASR_tokenizer_written_v2
• Task: Kazakh automatic speech recognition (Cyrillic script)
• Decoding: greedy CTC (no external language model)

Results

Evaluated with greedy CTC decoding. WER = Word Error Rate, CER = Unit Error Rate reported by the omnilingual-asr eval recipe (character-level). Lower is better.

Base → fine-tuned (WER):

The large improvement on Kazakh and the regression on English/Russian are the expected signature of monolingual specialisation (catastrophic forgetting). Russian degrades less than English, likely because KSC2 contains Russian loanwords and code-switching that keep some Russian signal alive during training.

Qualitative comparison (KSC2 test)

Sample transcriptions, base omniASR_CTC_300M_v2 vs this fine-tune, on the same audio. Picked to show typical base errors (word boundaries, vowels, endings) that fine-tuning fixes.

Limitations

• Monolingual specialisation. This model is intended for Kazakh only. On other languages it performs worse than the multilingual base model. Do not use it as a general multilingual ASR system.
• No language model. Reported numbers use greedy CTC decoding. Adding an n-gram LM (e.g. KenLM) with CTC beam search is expected to reduce WER further at inference time, with no retraining.
• Domain. Trained primarily on KSC2 (podcasts, radio, TV news, parliament, talk shows and crowdsourced readings). Far-field, heavy-noise or strongly accented speech outside this distribution may degrade.
• Audio format. Expects 16 kHz mono audio, amplitude-normalised, ≤ ~40 s per clip (split longer audio before transcription).
• Output format. Trained on normalised references (lowercased, whitespace-collapsed, no punctuation), so it emits lowercase Kazakh with no punctuation or capitalisation.

Usage

This is a fairseq2 / omnilingual-asr checkpoint, not a transformers model. Install the stack first:

# minimal — inference only:
pip install omnilingual-asr

# or the full pipeline stack (training, eval, augmentation):
#   pip install -r requirements.txt

# libsndfile is required for audio decoding:
#   Linux:  sudo apt-get install libsndfile1
#   macOS:  brew install libsndfile

Transcription goes through ASRInferencePipeline, which resolves a model asset card. Register a user card (in any dir exported as FAIRSEQ2_USER_ASSET_DIR) pointing at the downloaded omniASR_CTC_300M_kk.pt, then pass its name as model_card:

# cards/omniASR_CTC_300M_kk.yaml  (place in a dir on FAIRSEQ2_USER_ASSET_DIR)
name: omniASR_CTC_300M_kk
model_family: wav2vec2_asr
model_arch: 300m_v2
checkpoint: "/abs/path/to/omniASR_CTC_300M_kk.pt"
tokenizer_ref: omniASR_tokenizer_written_v2

from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

# CTC cards ignore the language argument, but the pipeline still expects the kwarg.
pipeline = ASRInferencePipeline(model_card="omniASR_CTC_300M_kk")

audio_files = ["kazakh_audio.flac"]   # 16 kHz mono; resampled automatically if needed
transcriptions = pipeline.transcribe(audio_files, lang=["kaz_Cyrl"], batch_size=1)
print(transcriptions[0])

Easiest reliable recipe: dump the base card with python -m fairseq2.assets show omniASR_CTC_300M_v2, copy all of its fields, and override only name + checkpoint. That pulls in tokenizer / tokenizer_family correctly.

Training

Fine-tuned with the omnilingual-asr wav2vec2.asr recipe (fairseq2 0.6). The recipe seeds the full model (encoder + CTC head) from the base checkpoint; the convolutional feature extractor stays frozen for stability.

• Base checkpoint: omniASR_CTC_300M_v2 (~300M params, full fine-tune)
• Tokenizer: omniASR_tokenizer_written_v2 (vocab 10288)
• Data: ISSAI KSC2 + Google FLEURS, Kazakh (kaz_Cyrl)
• Optimizer: AdamW, lr 1.5e-5, betas (0.9, 0.98), weight decay 0.0
• LR schedule: tri-stage, stage ratios [0.05, 0.35, 0.6], start scale 0.01, final scale 0.05
• Steps: 30,000
• Effective batch: ~36M audio elements/step (max_num_elements × grad_accum)
• Precision: bfloat16 (AMP)
• Hardware: 1 × NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM)
• Augmentation: 20% of training audio (KSC2 + FLEURS) replaced offline with augmented versions before parquet building — each clip gets a random mix of additive Gaussian noise, ±10% time-stretch, and room reverberation (audiomentations). Val/test left untouched. In-model latent masking stays off (the released CTC checkpoint has no masker module).

Launch (single config file, single GPU):

CUDA_VISIBLE_DEVICES=0 python -m workflows.recipes.wav2vec2.asr "$OUTPUT_DIR" \
  --config-file kk-config.yaml

Evaluation reproducibility

Numbers above were produced by transcribing each test set with greedy CTC decoding and computing WER/CER against the reference transcripts. KSC2 train/dev/test splits were prepared locally.

Train/test contamination was checked: 22% of test transcripts also occur in train (expected for read-speech, where prompts are re-read by different speakers), but 0 test clips share byte-identical audio with train — no recording leaked across the split. Speaker-level disjointness could not be verified because the parquet carries no speaker field. For a fully independent benchmark, evaluate on a public set that was not part of training — note that FLEURS Kazakh was folded into the training data here, so it does not qualify; Common Voice Kazakh test is a clean option.

License

Apache 2.0, inherited from the base Omnilingual ASR models and code.

Citation

If you use this model, please cite Omnilingual ASR and the KSC2 corpus.

@article{omnilingualasr2025,
  title  = {Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages},
  author = {{Omnilingual ASR Team}},
  journal = {arXiv preprint arXiv:2511.09690},
  year   = {2025}
}

@inproceedings{mussakhojayeva2022ksc2,
  title     = {KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus},
  author    = {Mussakhojayeva, Saida and Khassanov, Yerbolat and Varol, Huseyin Atakan},
  booktitle = {Interspeech},
  year      = {2022}
}