kannada-xeus-slam-asr

Kannada SLAM-ASR — xeus

Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + gemma-3-4b-it LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architecture comparison study (IIT Madras SPRING Lab).

Architecture

• Speech encoder (frozen): xeus (dim 1024)

• Projector: linear

• LLM: gemma-3-4b-it with LoRA (r=8, alpha=32, dropout=0.05) on q/k/v/o/gate/up/down_proj

PRELIMINARY run. eval_acc plateaued far below the other encoders (56%) -- superseded by xeus-v2-tomson-matched below (which fixed the setup to match Tomson's proven XEUS calling convention). Not decoded (no WER).

Result (epoch-1 boundary checkpoint, asr_epoch_1_step_29000)

No WER decode was run for this checkpoint (see note above).

pytorch_model.bin contains only the trainable weights (LoRA adapters + projector), not the frozen encoder(s) or base LLM weights.

Code

• code/finetune_data2vec_gemma3_kannada_xeus_cs.sh — training script

Core pipeline (SLAM-LLM, shared framework files)

core/ holds the underlying SLAM-ASR pipeline code the training/decode script above runs on top of -- a snapshot of the relevant files from SLAM-LLM-for-Kanada (a fork of ddlBoJack/SLAM-LLM) at the point this experiment was run.

Usage

This is an adapter/projector checkpoint, not a standalone model -- you need the core/ files above (or the full SLAM-LLM-for-Kanada codebase), the frozen speech encoder checkpoint(s) (/speech/abhishek/kannada_data/encoders/XEUS_repo/model/xeus_checkpoint_new.pth), and /speech/abhishek/SLAM_Hindi/models/google/gemma-3-4b-it to run inference. See code/finetune_data2vec_gemma3_kannada_xeus_cs.sh for the exact hydra config this checkpoint needs, and set ++ckpt_path=pytorch_model.bin.

Best checkpoint (highest eval_epoch_acc across the full training run)

best_checkpoint/pytorch_model.bin -- asr_epoch_1_step_28000 (eval_epoch_acc=56.46%, eval_epoch_loss=1.9558), found by scanning eval logs across the FULL training run (not just epoch-1). This is a LATER, slightly-better checkpoint than the epoch-1 boundary one above (same LoRA+projector weight format, just a different training step).