kannada-data2vec-ft-slam-asr

Kannada SLAM-ASR — data2vec-ft

Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + gemma-3-4b-it LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architecture comparison study (IIT Madras SPRING Lab).

Architecture

• Speech encoder (frozen): data2vec_aqc (dim 1024)

• Projector: linear

• LLM: gemma-3-4b-it with LoRA (r=8, alpha=32, dropout=0.05) on q/k/v/o/gate/up/down_proj

Baseline: single data2vec-AQC (Kannada fine-tuned) encoder + linear projector.

Result (epoch-1 boundary checkpoint, asr_epoch_1_step_33000)

pytorch_model.bin contains only the trainable weights (LoRA adapters + projector), not the frozen encoder(s) or base LLM weights.

Code

• code/finetune_data2vec_gemma3_kannada_cs.sh — training script
• code/inference_data2vec_gemma3_kannada_4testsets.sh — decode script (loops over 4 benchmark testsets)

Core pipeline (SLAM-LLM, shared framework files)

core/ holds the underlying SLAM-ASR pipeline code the training/decode script above runs on top of -- a snapshot of the relevant files from SLAM-LLM-for-Kanada (a fork of ddlBoJack/SLAM-LLM) at the point this experiment was run.

Usage

This is an adapter/projector checkpoint, not a standalone model -- you need the core/ files above (or the full SLAM-LLM-for-Kanada codebase), the frozen speech encoder checkpoint(s) (/speech/abhishek/kannada_data/encoders/SPRING_INX_data2vec_aqc_Kannada.pt), and /speech/abhishek/SLAM_Hindi/models/google/gemma-3-4b-it to run inference. See code/inference_data2vec_gemma3_kannada_4testsets.sh for the exact hydra config this checkpoint needs, and set ++ckpt_path=pytorch_model.bin.

Best checkpoint (highest eval_epoch_acc across the full training run)

best_checkpoint/pytorch_model.bin -- asr_epoch_1_step_32000 (eval_epoch_acc=88.03%, eval_epoch_loss=0.4795), found by scanning eval logs across the FULL training run (not just epoch-1). This is a LATER, slightly-better checkpoint than the epoch-1 boundary one above (same LoRA+projector weight format, just a different training step).