wav2vec2-large-zh-singing-phoneme-ctc

Wav2Vec2-Large-XLSR-53 — Singing Phoneme CTC Model (Two-Stage Fine-Tuning, LoRA Merged)

Model Summary

This model is fine-tuned from jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn via a two-stage training pipeline. The original character-level CTC head (~3,500 Chinese characters) is replaced with a custom multilingual phoneme head (POLYGLOT, 39 phonemes), targeting phoneme-level temporal alignment of singing vocals (e.g., karaoke lyrics synchronization).

This checkpoint is a fully merged model. The Stage 2 LoRA adapter weights have been merged into the base model via merge_and_unload(). No PEFT dependency is required at inference time — load directly with transformers.

---

Model Architecture

• Backbone: Wav2Vec2-Large (XLSR-53), encoder hidden size 1024
• CTC Head: Dropout(0.1) + Linear(1024 → 39) (replaces the original Chinese character CTC head)
• Stage 2 LoRA Adapter (merged): Originally applied to all attention and FFN layers in the encoder (see hyperparameters below); weights have been merged into the backbone via merge_and_unload()

---

Phoneme Set (POLYGLOT — 39 tokens)

The phoneme set is designed to cover Mandarin Chinese (pinyin initial/final decomposition), English (ARPAbet mapping), and a subset of Japanese phonemes.

Note — Training coverage: Although the vocabulary is multilingual by design (zh / en / ja), all training data in both stages is Mandarin-only (AISHELL-3 + M4Singer). The English and Japanese phonemes (v, th, dh, hy, jh, dz, ü, ae, ir, …) exist in the vocabulary but have received no supervised signal; their behaviour at inference time is untested and likely unreliable.

---

Training Procedure

Stage 1 — Frozen Encoder, New CTC Head (AISHELL-3)

Stage 2 — LoRA Fine-Tuning (M4Singer)

---

Usage

Load the Model

This checkpoint is already merged — load it directly with transformers, no PEFT required.

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import torch

processor = Wav2Vec2Processor.from_pretrained("path/to/final")
model = Wav2Vec2ForCTC.from_pretrained("path/to/final")
model.eval()

Inference (CTC Greedy Decode)

import torchaudio

waveform, sr = torchaudio.load("singing.wav")
if sr != 16000:
    waveform = torchaudio.functional.resample(waveform, sr, 16000)
waveform = waveform.mean(0)  # convert to mono

inputs = processor(waveform, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits  # (1, T, 39)

# CTC greedy decode: argmax → collapse repeats → remove blank (id=0)
pred_ids = logits.argmax(dim=-1)[0].tolist()
decoded = []
prev = -1
for idx in pred_ids:
    if idx != prev:
        if idx != 0:
            decoded.append(idx)
        prev = idx

id2phoneme = {v: k for k, v in processor.tokenizer.get_vocab().items()}
phonemes = [id2phoneme[i] for i in decoded]
print(phonemes)

Merge LoRA Weights

Already done. This checkpoint was saved after calling merge_and_unload(). The code below is provided for reference only, in case you are working from an unmerged PEFT checkpoint.

# Only needed if starting from a raw PEFT checkpoint (not this one)
from peft import PeftModel
base_model = Wav2Vec2ForCTC.from_pretrained("path/to/stage1_best")
peft_model = PeftModel.from_pretrained(base_model, "path/to/stage2_peft")
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("path/to/final")
processor.save_pretrained("path/to/final")

---

Intended Use

• Singing vocal alignment: Force-aligning phoneme sequences to audio timestamps (CTC forced alignment) for generating per-character / per-phoneme timing (.lrc, .srt, .ass)
• Singing acoustics research: Phoneme-level feature extraction and articulation analysis for Mandarin singing

Out-of-Scope Use

• General Mandarin ASR: Use the original jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn instead
• Non-Mandarin singing: The phoneme set supports multilingual phonemes by design, but training data is Mandarin-only; generalization to other languages is unverified
• Real-time / low-latency applications: The model is not optimized for streaming inference

---

Training Datasets

---

Limitations

• The model outputs a phoneme sequence, not Chinese characters; it cannot be used for text transcription without a phoneme-to-pinyin/hanzi conversion step
• Trained primarily on Mandarin singing (M4Singer); generalization to other styles (e.g., Cantonese, English pop, rap) may be limited
• The custom phoneme vocabulary requires the accompanying tokenizer (vocab.json) for decoding

---

Framework Versions

• Transformers (compatible with PEFT 0.18.1)
• PyTorch (with bfloat16 AMP support)
• PEFT 0.18.1 (training only; not required at inference)