sushrota-sanskrit-asr

Su-śrotā — Sanskrit ASR (finetuned IndicConformer-CTC)

A Sanskrit speech-recognition model finetuned for śāstric and recitational Sanskrit (chant and prose), where general-purpose Sanskrit ASR degrades badly. It is the model behind Vāgbodhinī and Su-śrotā, live Sanskrit chant-practice and dictation tools.

• Base: AI4Bharat IndicConformer (EncDecHybridRNNTCTCBPEModel, ~129 M params). We use the CTC head on the Sanskrit token slice.
• Author: Prof. Prathosh A P, Indian Institute of Science, Bengaluru.

Checkpoints in this repo

Results

The headline gain of the current model is real-world robustness. Evaluated on a leakage-free, 327-clip held-out set of in-the-wild user recordings (phones, rooms, varied speakers):

On curated/studio domains, v13b preserves the base model's quality while adding that robustness:

Note on WER vs SN-WER: Sanskrit word boundaries are largely orthographic (sandhi fuses words), so raw WER over-penalises boundary disagreements. Roughly half of the WER is spacing; CER and SN-WER (sandhi-normalised) are the meaningful numbers.

Training Data

The deployed checkpoint (v13b) was trained on 17.4 hours / 6,438 utterances, combining curated scholar recordings with consented, quality-tiered in-the-wild data.

Split: ~90% curated/scholar (15.65 h) + ~10% consented in-the-wild (1.73 h).

Data pipeline & quality tiering

In-the-wild audio is collected with consent through the practice tools (users read a known reference text) and automatically quality-graded at the akṣara level against that reference. Each clip is tiered — pass / override (clean, trainable), review (partial match), or low / unclear (archived). Only the clean tiers enter training.

The auto-grader was audited for reliability: all quarantined review clips were re-decoded with the improved model and promoted back to gold only when they scored a perfect akṣara match. Just 4.3% proved to be model error — confirming the remaining ~96% are genuine reader deviations (mispronunciations, disfluencies, noise), i.e. the grader correctly quarantines real problems rather than mere model disagreements.

Consented corpus

Beyond what is trained on, the flywheel has banked ~28 hours of consented Sanskrit audio from ~1,500 speakers across 11 input scripts (Devanāgarī, Kannada, Telugu, IAST, and others) — a growing resource for future, more speaker- and script-diverse training rounds.

Usage

import json, numpy as np, torch, soundfile as sf
import nemo.collections.asr as na

M = na.models.EncDecHybridRNNTCTCBPEModel.restore_from("sushrota_sanskrit_asr_v13b.nemo").eval()

OFF, V, BLANK = 4096, 256, 5632          # Sanskrit token slice of the aggregate vocab
def greedy(wav):                          # wav: 16 kHz mono float32
    sig = torch.tensor(wav).unsqueeze(0); sl = torch.tensor([len(wav)])
    with torch.no_grad():
        enc, _ = M.forward(input_signal=sig, input_signal_length=sl)
        lp = M.ctc_decoder(encoder_output=enc)[0].cpu().numpy()
    cols = [BLANK] + list(range(OFF, OFF + V))
    P = lp[:, cols]; P = P - (P.max(1, keepdims=True) +
        np.log(np.exp(P - P.max(1, keepdims=True)).sum(1, keepdims=True)))  # re-log_softmax on slice
    ids = P.argmax(1)
    sub = M.tokenizer.tokenizers_dict["sa"]                 # Sanskrit SentencePiece
    out, prev = [], -1
    for i in ids:
        i = int(i)
        if i != prev and i != 0: out.append(sub.ids_to_tokens([i - 1])[0])
        prev = i
    return "".join(out).replace("▁", " ").strip()

wav, sr = sf.read("clip.wav", dtype="float32")
print(greedy(wav))

The model is an aggregate multilingual IndicConformer; decode on the Sanskrit slice as above.

Provenance & consent

Scholar recordings were contributed for the purpose of building this model. In-the-wild clips are collected only with explicit user consent and store no raw IP or personal identifiers (an anonymous per-session id only). Individual reciter names are not published.

Related

• Vāgbodhinī (chant-practice tool + full experiment report): https://github.com/prathoshap/sushrota-sanskrit-asr
• Vāgdhenu (metre-aware Sanskrit chant TTS): prathoshap/vagdhenu

License

Finetuned from AI4Bharat's IndicConformer — please observe the base model's license terms.

Citation

Prathosh A P, Su-śrotā: Scholar-grade Sanskrit ASR and metre-aware chant practice, Indian Institute of Science, Bengaluru, 2026.