kannada-dual-encoder-slam-asr

Kannada SLAM-ASR — dual-encoder

Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + gemma-3-4b-it LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architecture comparison study (IIT Madras SPRING Lab).

Architecture

• Speech encoders (both frozen): data2vec-AQC (Kannada fine-tuned) + data2vec-AQC (SSL pretrained), same raw audio, concat-fused (dim 2048)

• Projector: linear (fused-dim)

• LLM: gemma-3-4b-it with LoRA (r=8, alpha=32, dropout=0.05) on q/k/v/o/gate/up/down_proj

Dual-encoder concat-fusion experiment: data2vec-AQC (Kannada fine-tuned) + data2vec-AQC (SSL pretrained), both frozen, both fed the SAME raw audio, fused by feature-dim concatenation (1024+1024=2048) after temporal alignment, then the EXISTING linear projector re-instantiated with the doubled input dim. Best epoch-1 result of all Kannada encoder experiments here.

Result (epoch-1 boundary checkpoint, asr_epoch_1_step_33000)

WER (4 benchmark testsets)

pytorch_model.bin contains only the trainable weights (LoRA adapters + projector), not the frozen encoder(s) or base LLM weights.

Code

• code/finetune_data2vec_gemma3_kannada_dualencoder_cs.sh — training script
• code/inference_data2vec_gemma3_kannada_dualencoder_4testsets.sh — decode script (loops over 4 benchmark testsets)

Core pipeline (SLAM-LLM, shared framework files)

core/ holds the underlying SLAM-ASR pipeline code the training/decode script above runs on top of -- a snapshot of the relevant files from SLAM-LLM-for-Kanada (a fork of ddlBoJack/SLAM-LLM) at the point this experiment was run.

Usage

This is an adapter/projector checkpoint, not a standalone model -- you need the core/ files above (or the full SLAM-LLM-for-Kanada codebase), the frozen speech encoder checkpoint(s) (data2vec-AQC Kannada-FT + data2vec-AQC SSL, both SPRING_INX checkpoints), and /speech/abhishek/SLAM_Hindi/models/google/gemma-3-4b-it to run inference. See code/inference_data2vec_gemma3_kannada_dualencoder_4testsets.sh for the exact hydra config this checkpoint needs, and set ++ckpt_path=pytorch_model.bin.

Best checkpoint (highest eval_epoch_acc across the full training run)

best_checkpoint/pytorch_model.bin -- asr_epoch_1_step_32000 (eval_epoch_acc=88.37%, eval_epoch_loss=0.4648), found by scanning eval logs across the FULL training run (not just epoch-1). This is a LATER, slightly-better checkpoint than the epoch-1 boundary one above (same LoRA+projector weight format, just a different training step).