wav2vec-ug-finetuned-1b

wav2vec2-mms-1b-uyghur-latin-ctc

A Uyghur (Latin transliteration) automatic speech recognition model, fine-tuned from ixxan/wav2vec2-large-mms-1b-uyghur-latin using an output-head-only adaptation strategy.

Character Error Rate: 0.0517 — approximately one incorrect character in every nineteen.

The model was adapted by freezing all 964M backbone parameters and retraining only the CTC output projection (lm_head) — 46,116 trainable parameters, 0.005% of the model — for a single epoch on 2 × Tesla T4 GPUs.

---

Model description

Adaptation strategy

The base checkpoint had already been fine-tuned on Uyghur Latin speech, so its acoustic encoder was already correct for the target language. The only component that did not transfer was the output alphabet: the base checkpoint exposes a 34-token vocabulary, while the target corpus requires 36.

Rather than fine-tune the full network, the original Linear(1280 → 34) head was replaced with a freshly initialised Linear(1280 → 36), and everything else in the network was frozen. The head therefore only has to learn the projection from an already-correct acoustic representation onto a new symbol set.

This converges extremely quickly — training loss fell from 12.51 to below 1.0 within 50 optimizer steps — which empirically confirms that the frozen encoder's frame representations were already close to linearly separable into the target characters.

Consequences of freezing:

• No Adam optimizer state for 964M parameters (≈7 GB of GPU memory saved)
• No backward pass through 48 Transformer layers
• No risk of catastrophic forgetting, which permits an aggressive 1e-3 learning rate that would be unsafe on an unfrozen backbone

---

Training specification

Hardware

Hyperparameters

Training data

Training loss

---

Evaluation

CER is the Levenshtein edit distance between prediction and reference, normalised by reference length, computed with jiwer via the evaluate library.

---

Usage

Installation

pip install transformers torch torchaudio soundfile librosa

Quick start

import torch, librosa
from transformers import AutoModelForCTC, Wav2Vec2Processor

MODEL_ID = "Shramadeepd/wav2vec-ug-finetuned-1b"  # replace with your repo id

processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID, torch_dtype=torch.float16).to("cuda").eval()

# Audio MUST be mono 16 kHz — the CNN feature encoder assumes 320 samples per 20 ms frame
speech, _ = librosa.load("sample.wav", sr=16000, mono=True)

inputs = processor(speech, sampling_rate=16000, return_tensors="pt")
inputs = {k: (v.to("cuda", torch.float16) if v.is_floating_point() else v.to("cuda"))
          for k, v in inputs.items()}

with torch.no_grad():
    logits = model(**inputs).logits

pred_ids = torch.argmax(logits, dim=-1)
print(processor.batch_decode(pred_ids)[0])

Batched inference

from torch.utils.data import DataLoader

def collate(batch):
    return processor.pad([{"input_values": x} for x in batch],
                         padding=True, return_tensors="pt")

loader = DataLoader(waveforms, batch_size=4, collate_fn=collate, shuffle=False)

transcriptions = []
for batch in loader:
    batch = {k: (v.to("cuda", torch.float16) if v.is_floating_point() else v.to("cuda"))
             for k, v in batch.items()}
    with torch.no_grad():
        logits = model(**batch).logits
    transcriptions += processor.batch_decode(torch.argmax(logits, dim=-1))

CPU inference

Drop torch_dtype=torch.float16 and the .to("cuda") calls. Expect roughly 10–20× slower inference; fp16 is not well supported on CPU.

---

Continuing training / further fine-tuning

The checkpoint ships with its processor, so it can be used as a starting point directly.

Option A — continue head-only training

Cheapest path; useful if you have more data from a similar acoustic domain.

from transformers import AutoModelForCTC, Wav2Vec2Processor

processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID, ctc_loss_reduction="mean",
                                        pad_token_id=processor.tokenizer.pad_token_id)

model.freeze_feature_encoder()
for name, param in model.named_parameters():
    if "lm_head" not in name:
        param.requires_grad = False

Option B — unfreeze the upper encoder layers

Recommended next step for pushing CER below 0.05. Use a much lower learning rate (1e-5 to 3e-5); the 1e-3 used for the random head will destroy pretrained weights.

model.freeze_feature_encoder()
for name, param in model.named_parameters():
    param.requires_grad = any(k in name for k in ["lm_head"] +
                              [f"encoder.layers.{i}." for i in range(40, 48)])

Enable gradient_checkpointing_enable() in this mode — with real backpropagation through the Transformer, the memory/compute trade becomes worthwhile on 14 GB cards.

Option C — adapt to a different alphabet

If your transcripts use a different character set (e.g. Perso-Arabic Uyghur script), build a new vocabulary from your corpus, construct a new Wav2Vec2CTCTokenizer, and replace lm_head again:

import torch.nn as nn
model.lm_head = nn.Linear(model.config.hidden_size, new_processor.tokenizer.vocab_size)
model.config.vocab_size = new_processor.tokenizer.vocab_size

---

Limitations and intended use

• Domain. Trained on read/broadcast-style Uyghur speech. Expect degradation on spontaneous conversation, heavy accents, overlapping speakers, telephony-band audio, or noisy field recordings.
• Script. Outputs Latin transliteration only, with case-sensitive characters (A G H J N O U are distinct phonemes, not capitalisation). It does not produce Perso-Arabic Uyghur script. Never lowercase the output.
• No punctuation or casing restoration. The training transcripts contain no punctuation or digits, so neither is ever emitted.
• Greedy decoding. No language model is used. Adding KenLM beam search via pyctcdecode typically yields a further 10–20% relative CER reduction.
• Input contract. Audio must be mono and resampled to 16 kHz. Any other sample rate produces silently wrong output, not an error.
• Frozen backbone. Since only the head was trained, the model inherits all biases and domain characteristics of the MMS-1B base checkpoint.

This model is released for research and educational use. It should not be used as the sole component in high-stakes settings (legal, medical, surveillance) without human review.

---

Citation

If you use this model, please cite the underlying work it builds on:

@article{pratap2023mms,
  title   = {Scaling Speech Technology to 1,000+ Languages},
  author  = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and others},
  journal = {arXiv preprint arXiv:2305.13516},
  year    = {2023}
}

@article{baevski2020wav2vec,
  title   = {wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations},
  author  = {Baevski, Alexei and Zhou, Henry and Mohamed, Abdelrahman and Auli, Michael},
  journal = {NeurIPS},
  year    = {2020}
}

Acknowledgements

• Base checkpoint: ixxan/wav2vec2-large-mms-1b-uyghur-latin
• Massively Multilingual Speech (MMS) project, Meta AI
• NPPE-2 Uyghur ASR Challenge organisers