wav2vec2-vedic-aligner

Vedic chanting aligner (wav2vec2-CTC, Devanagari)

addy88/wav2vec2-sanskrit-stt fine-tuned on Vedavani (43.5 h of chanted Sanskrit) for forced alignment of a known Devanagari transcript onto a recitation.

It is used as the alignment and pronunciation backbone of a Vedic chanting assessment app: syllable boundaries come from torchaudio.forced_align over its emissions, and pronunciation is Goodness-of-Pronunciation off its posteriors. It is trained to place chanted Sanskrit accurately, not to be a general-purpose Sanskrit ASR system.

Two things you must know before using it

1. The CTC blank is <s> (id 0), not <pad>. This is inherited from the base checkpoint, whose config names <pad> while the token that actually behaves as the blank is <s> -- it argmaxes on ~89 % of frames while <pad> never fires. Pass blank=0 to torchaudio.forced_align, and strip <s> yourself when decoding: processor.batch_decode strips the nominal blank and will leave literal <s> in every hypothesis.

2. Speed perturbation is why the boundaries hold. Vedavani is single verses at one tempo. Without augmentation the model places the elongated final syllable of a chanted line correctly at 1.0x and truncates it past 1.06x -- which reads as a much worse rhythm measurement, not as a worse loss. Trained here with resampling over 0.85-1.25x.

Results

Training

12 epochs, lr 1e-4 linear with 500 warmup steps, effective batch 24, fp16, feature encoder frozen, ctc_loss_reduction="mean", clips capped at 16 s, speed perturbation 0.85-1.25x. The vocab is reused byte for byte from the base checkpoint -- 67 tokens, never rebuilt.

Usage

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

processor = Wav2Vec2Processor.from_pretrained("chiranjeevisagi/wav2vec2-vedic-aligner")
model = Wav2Vec2ForCTC.from_pretrained("chiranjeevisagi/wav2vec2-vedic-aligner")
# ... and remember blank = <s> = id 0

License

Apache-2.0, matching the base checkpoint and the corpus.