Downloads · 30 days
0
Abhisingh-18/kannada-sarvam1-slam-asr
kannada-sarvam1-slam-asr is a machine learning model from Abhisingh-18. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + sarvam-1 LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architect…
Downloads · 30 days
0
Access
Public
Updated Sep 23, 2026
Repo size
213 MB
Likes
0
Public
Click a slice to open those files.
.bin213 MB · 100%
From the Hugging Face model README
Bilingual Kannada-English speech recognition using a frozen speech encoder + projector + sarvam-1 LLM decoder fine-tuned with LoRA (SLAM-ASR architecture). One experiment from Prof. Umesh S's Kannada encoder/architecture comparison study (IIT Madras SPRING Lab).
Speech encoder (frozen): data2vec_aqc (dim 1024)
Projector: linear
LLM: sarvam-1 with LoRA (r=8, alpha=32, dropout=0.05) on q/k/v/o/gate/up/down_proj
LLM-swap experiment: same data2vec-AQC (Kannada FT) encoder + linear projector as the baseline, but the LLM decoder is Sarvam-1 (LlamaForCausalLM, 2.5B) instead of Gemma-3-4B-IT. Required a real bugfix in slam_model_new.py's setup_llm(): the generic AutoModelForCausalLM path was passing use_cache=None/load_in_8bit=None/device_map=None, which a newer transformers/huggingface_hub rejects -- fixed by omitting unset kwargs instead of passing None (first surfaced by this experiment; every prior one here used Gemma, whose special-cased loader already guarded this).
asr_epoch_1_step_33000)| Metric | Value |
|---|---|
| eval_epoch_acc (teacher-forced, held-out dev split) | 85.90% |
| eval_epoch_loss | 0.5771 |
| Testset | WER |
|---|---|
| FLEURS | 22.83% |
| IndicTTS | 21.26% |
| Kathbath | 18.45% |
| Kathbath-Noisy | 20.23% |
pytorch_model.bin contains only the trainable weights (LoRA adapters +
projector), not the frozen encoder(s) or base LLM weights.
code/finetune_data2vec_sarvam1_kannada_cs.sh — training scriptcode/inference_data2vec_sarvam1_kannada_4testsets.sh — decode script (loops over 4 benchmark testsets)core/ holds the underlying SLAM-ASR pipeline code the training/decode
script above runs on top of -- a snapshot of the relevant files from
SLAM-LLM-for-Kanada
(a fork of ddlBoJack/SLAM-LLM) at
the point this experiment was run.
This is an adapter/projector checkpoint, not a standalone model -- you need
the core/ files above (or the full
SLAM-LLM-for-Kanada
codebase), the frozen speech encoder checkpoint(s) (/speech/abhishek/kannada_data/encoders/SPRING_INX_data2vec_aqc_Kannada.pt), and
/speech/tomson/exps/speech-recog/models/sarvam-1 to run inference. See code/inference_data2vec_sarvam1_kannada_4testsets.sh for the exact
hydra config this checkpoint needs, and set ++ckpt_path=pytorch_model.bin.
best_checkpoint/pytorch_model.bin -- asr_epoch_1_step_32000 (eval_epoch_acc=85.95%, eval_epoch_loss=0.5787), found by scanning eval logs across the FULL training run (not just epoch-1). This is a LATER, slightly-better checkpoint than the epoch-1 boundary one above (same LoRA+projector weight format, just a different training step).