Longform-ASR-for-Indian-Laungwage

Longform ASR for Indian Languages

Automatic Speech Recognition (ASR) for Indian languages (Tamil & Hindi) using the SLAM-LLM framework with Silero VAD-based longform inference and rolling previous context.

---

Architecture

Raw Audio
    │
    ▼
┌─────────────────────────────────────┐
│   Speech Encoder: data2vec-AQC  ❄️   │  ← FROZEN
│   Params : 314.32 M                 │
│   Trainable : 0.00 M               │
└─────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────┐
│   Projector: Linear             🔥   │  ← TRAINABLE
│   Params : 14.68 M                  │
└─────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────┐
│   LLM Decoder: Sarvam-1         🔥   │  ← TRAINABLE (LoRA)
│   Total Params  : 2.52 B            │
│   Trainable     : 11.98 M           │
│   LoRA rank     : 32                │
│   LoRA alpha    : 128               │
│   LoRA targets  : q/k/v/o/gate/     │
│                   up/down proj      │
└─────────────────────────────────────┘
    │
    ▼
  Text Transcription

Parameter Summary

---

Training

Setup

• Data: Tamil (~2000 hours), Hindi (~4000 hours)
• Encoder: SPRING_INX_data2vec_aqc_Tamil.pt / SPRING_INX_data2vec_aqc_Hindi.pt (frozen)
• Projector: Linear layer (5x downsampling rate)
• LLM: Sarvam-1 with LoRA fine-tuning
• Training framework: DeepSpeed multi-GPU

LoRA Config

r           = 32
lora_alpha  = 128
dropout     = 0.05
bias        = none
target      = q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Prompt (with previous context)

Using previous context: {prev_context}. Transcribe speech to text. Apply relevant details from the previous context.

Training Scripts

# Tamil
bash examples/asr_librispeech/scripts/finetune_data2vec_sarvam1_longform.sh

# Hindi
bash examples/asr_librispeech/scripts/finetune_data2vec_sarvam1_longform_hindi.sh

---

Inference (Longform)

Pipeline

Long Audio File
      │
      ▼
 [Step 1] ffmpeg → 16kHz mono WAV
      │
      ▼
 [Step 2] Silero VAD chunking
          threshold     = 0.4
          max chunk     = 15 seconds
          min chunk     = 1 second
      │
      ▼
 [Step 3] Chunk-by-chunk inference
          with 1 previous chunk context
      │
      ▼
 Transcription (tab-separated: key \t text)

Run Inference

# Tamil
bash examples/asr_librispeech/scripts/run_longform_tamil_1ctx.sh \
    /path/to/audio.wav \
    my_experiment

# Hindi
bash examples/asr_librispeech/scripts/run_longform_hindi_1ctx.sh \
    /path/to/audio.wav \
    my_experiment

Output at: /speech/abhishek/exps/longform/<experiment_name>/decode_1ctx/decode_pred

---

Tamil ASR Results (WER %)

Model Comparison

Test Set Details

Best overall: Sarvam-1 wins on CommonVoice, FLEURS, MUCS, MILE, r1_eval, r2_eval, MIC

---

Hindi ASR Results (WER %)

Model Comparison

Test Set Details

Training Data: Hindi ~4000 Hours
Best overall: Sarvam-1 wins on CommonVoice, IndicTTS, Kathbath, Kathbath Noisy, MUCS

---

Checkpoints

---

Requirements

pip install -r requirements.txt

• Python 3.10
• PyTorch + CUDA
• Silero VAD (torch.hub)
• HuggingFace Transformers
• DeepSpeed (for training)
• ffmpeg