wav2vec2-large-xls-r-300m-Urdu

Urdu ASR XLS-R 300M

A fine-tuned XLS-R 300M CTC model for Urdu automatic speech recognition. It transcribes 16 kHz mono audio and includes an optional 5-gram KenLM decoder.

Best reported result: 39.89% WER / 16.70% CER with KenLM decoding on the Urdu Common Voice 8.0 test set. See the Kaggle evaluation notebook for a reproducible example.

⚡ Quick start

Install the required packages:

pip install -U torch torchaudio transformers pyctcdecode kenlm huggingface_hub

Note: After installing the packages in a notebook environment, restart the kernel before running the inference code.

Transcribe a local audio file:

import torch
import torchaudio
from transformers import AutoModelForCTC, AutoProcessor

MODEL_ID = "kingabzpro/wav2vec2-large-xls-r-300m-Urdu"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
INFERENCE_DTYPE = torch.float16 if DEVICE.type == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID).eval().to(
    device=DEVICE, dtype=INFERENCE_DTYPE
)

waveform, sample_rate = torchaudio.load("audio.wav")

# Convert stereo (or multi-channel) audio to mono and resample to 16 kHz.
waveform = waveform.mean(dim=0)
if sample_rate != 16_000:
    waveform = torchaudio.functional.resample(waveform, sample_rate, 16_000)

inputs = processor(
    waveform.numpy(), sampling_rate=16_000, return_tensors="pt", padding=True
).input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)

with torch.inference_mode():
    predicted_ids = model(inputs).logits.argmax(dim=-1)

transcription = processor.batch_decode(predicted_ids)[0]
print(transcription)

🧠 Language-model decoding

Why use it? The included 5-gram KenLM language model reduces the reported full-test WER from 56.07% (greedy CTC) to 39.89%.

<details> <summary><strong>Show the complete KenLM decoding example</strong></summary> <br>

The repository contains a 5-gram KenLM language model.

import json
import torch
import torchaudio
from huggingface_hub import hf_hub_download
from pyctcdecode import build_ctcdecoder
from transformers import AutoModelForCTC, AutoProcessor

MODEL_ID = "kingabzpro/wav2vec2-large-xls-r-300m-Urdu"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
INFERENCE_DTYPE = torch.float16 if DEVICE.type == "cuda" else torch.float32

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID).eval().to(
    device=DEVICE, dtype=INFERENCE_DTYPE
)

kenlm_path = hf_hub_download(MODEL_ID, "language_model/5gram.bin")
unigrams_path = hf_hub_download(MODEL_ID, "language_model/unigrams.txt")
attrs_path = hf_hub_download(MODEL_ID, "language_model/attrs.json")

with open(unigrams_path, encoding="utf-8") as file:
    unigrams = [line.strip() for line in file if line.strip()]
with open(attrs_path, encoding="utf-8") as file:
    attrs = json.load(file)

# Keep only acoustic tokens. The matching ID list is then applied to logits.
vocab_items = sorted(processor.tokenizer.get_vocab().items(), key=lambda item: item[1])
blank_id = processor.tokenizer.pad_token_id
delimiter = processor.tokenizer.word_delimiter_token
decoder_pairs = [
    (token, token_id)
    for token, token_id in vocab_items
    if token_id == blank_id or token == delimiter or len(token) == 1
]
kept_token_ids = [token_id for _, token_id in decoder_pairs]
labels = [
    "" if token_id == blank_id else " " if token == delimiter else token
    for token, token_id in decoder_pairs
]
decoder = build_ctcdecoder(
    labels,
    kenlm_model_path=kenlm_path,
    unigrams=unigrams,
    alpha=attrs.get("alpha", 0.5),
    beta=attrs.get("beta", 1.0),
)

waveform, sample_rate = torchaudio.load("audio.wav")
waveform = waveform.mean(dim=0)
if sample_rate != 16_000:
    waveform = torchaudio.functional.resample(waveform, sample_rate, 16_000)

inputs = processor(
    waveform.numpy(), sampling_rate=16_000, return_tensors="pt"
).input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)
with torch.inference_mode():
    logits = model(inputs).logits[0].float().cpu().numpy()

transcription = decoder.decode(logits[:, kept_token_ids])
print(transcription)

</details>

🧪 Kaggle evaluation

The Kaggle notebook evaluates a five-sample streaming smoke test from fixie-ai/common_voice_17_0 (ur, test).

<details> <summary><strong>Show the core evaluation code</strong></summary> <br>

from datasets import Audio, load_dataset

stream = load_dataset(
    "fixie-ai/common_voice_17_0", "ur", split="test", streaming=True
).cast_column("audio", Audio(sampling_rate=16_000))

example = next(iter(stream))
audio = example["audio"]
samples = audio.get_all_samples().data if hasattr(audio, "get_all_samples") else audio["array"]
waveform = samples.detach().cpu().numpy() if torch.is_tensor(samples) else samples
if waveform.ndim == 2:
    waveform = waveform.mean(axis=0 if waveform.shape[0] <= waveform.shape[-1] else 1)

inputs = processor(waveform, sampling_rate=16_000, return_tensors="pt")
with torch.inference_mode():
    logits = model(inputs.input_values.to(device=DEVICE, dtype=INFERENCE_DTYPE)).logits[0]

prediction = decoder.decode(logits.float().cpu().numpy()[:, kept_token_ids])
print("Reference: ", example["sentence"])
print("Prediction:", prediction)

</details>

Recorded notebook output

Single-sample inference:

Reference:  بے ذوق نہیں اگرچہ فطرت
Prediction: بھی ذوق نہیں اگھرچے فطرت

Five-sample streaming smoke-test results:

Important: This is a five-sample smoke test—not a benchmark. Do not compare it directly with the full Common Voice 8.0 test-set results below.

📊 Evaluation

Full Common Voice 8.0 test set

Results are reported on the Urdu test split of Mozilla Common Voice 8.0. The language-model row is the model-card score; compare each result only with the same decoding strategy.

To reproduce language-model evaluation from this repository:

python eval.py --model_id kingabzpro/wav2vec2-large-xls-r-300m-Urdu --dataset mozilla-foundation/common_voice_8_0 --config ur --split test

🏗️ Training

The model was trained from facebook/wav2vec2-xls-r-300m on Urdu Mozilla Common Voice 8.0.

<details> <summary><strong>Training checkpoints</strong></summary> <br> </details>

⚠️ Intended use and limitations

Use this model for Urdu speech transcription and prototyping. Accuracy varies with recording quality, speaker accent, code-switching, background noise, domain-specific vocabulary, and utterance length. Review transcripts before using them in consequential or user-facing workflows.

<details> <summary><strong>Historical training environment</strong></summary> <br>

• Transformers 4.17.0.dev0
• PyTorch 1.10.2+cu102
• Datasets 1.18.2.dev0
• Tokenizers 0.11.0

</details>