kannada-w2vbert-slam-asr

Kannada SLAM-ASR — w2vbert

Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + gemma-3-4b-it LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architecture comparison study (IIT Madras SPRING Lab).

Architecture

• Speech encoder (frozen): wav2vec2_bert (dim 1024)

• Projector: linear

• LLM: gemma-3-4b-it with LoRA (r=8, alpha=32, dropout=0.05) on q/k/v/o/gate/up/down_proj

PRELIMINARY run. eval_acc plateaued far below the other encoders (57%) -- superseded by the w2vbert-v2-fresh re-run below. Kept here for the full experiment record; not decoded (no WER).

Result (epoch-1 boundary checkpoint, asr_epoch_1_step_27000)

No WER decode was run for this checkpoint (see note above).

pytorch_model.bin contains only the trainable weights (LoRA adapters + projector), not the frozen encoder(s) or base LLM weights.

Code

• code/finetune_data2vec_gemma3_kannada_w2vbert_cs.sh — training script

Core pipeline (SLAM-LLM, shared framework files)

core/ holds the underlying SLAM-ASR pipeline code the training/decode script above runs on top of -- a snapshot of the relevant files from SLAM-LLM-for-Kanada (a fork of ddlBoJack/SLAM-LLM) at the point this experiment was run.

Usage

This is an adapter/projector checkpoint, not a standalone model -- you need the core/ files above (or the full SLAM-LLM-for-Kanada codebase), the frozen speech encoder checkpoint(s) (/speech/abhishek/kannada_data/encoders/wav2vec2-bert-2.0), and /speech/abhishek/SLAM_Hindi/models/google/gemma-3-4b-it to run inference. See code/finetune_data2vec_gemma3_kannada_w2vbert_cs.sh for the exact hydra config this checkpoint needs, and set ++ckpt_path=pytorch_model.bin.

Best checkpoint (highest eval_epoch_acc across the full training run)

The epoch-1 boundary checkpoint uploaded above (asr_epoch_1_step_27000, eval_epoch_acc=57.12%) IS the best checkpoint across the whole logged training run -- no separate file needed.