hindi-english-codeswitch-asr

Hindi-English Code-Switch ASR (data2vec-AQC + Gemma-3-4B-IT LoRA)

A bilingual automatic speech recognition (ASR) model for Hindi, English, and Hindi-English code-switched speech, built on the SLAM-LLM framework. It combines a frozen data2vec-AQC speech encoder, a trainable linear projector, and google/gemma-3-4b-it fine-tuned with LoRA.

This checkpoint (epoch2_step146286) was selected as the overall best checkpoint by validation loss across the full training run, and independently confirmed as the strongest or most balanced checkpoint across 9 held-out Hindi/code-switch benchmark testsets (6 out of 9 wins vs. neighboring checkpoints — see comparison table below).

Architecture

Only the projector + LoRA adapter weights are released here (~117MB) — the base data2vec-AQC encoder and google/gemma-3-4b-it weights are not included and must be obtained separately (see Usage below).

Training Data

• 2,683,431 utterances total from a bilingual Hindi-English code-switch corpus
• ~4000 hours Hindi speech, ~1000 hours English speech, with code-switch (hi-en) tagged utterances
• Prompt format follows Gemma's chat template with a rolling previous-utterance-context field for long-form continuity

Checkpoint Comparison (WER %, lower is better)

WER computed on 9 held-out Hindi/code-switch benchmark testsets, using an edit-distance-based scorer with NFC normalization and punctuation stripped from both reference and hypothesis (standard WER convention).

This checkpoint (Epoch 2, step 146286) wins 6 of 9 testsets and is the most balanced overall.

Comparison vs. Meta's Omnilingual ASR (omniASR_LLM_7B_v2)

This model (a specialized ~30M-trainable-parameter bilingual fine-tune) outperforms Meta's general-purpose 7B multilingual ASR model on 8 of 9 Hindi/code-switch benchmarks:

Note: Omnilingual ASR has no dedicated Hindi-English code-switch language code — each utterance is forced into a single target script (hin_Deva or eng_Latn), which likely disadvantages it on the code-switch-heavy testsets (Vendor R1/R2) shown above. This model was trained specifically for code-switch conditioning.

Usage

This is not a plain transformers-loadable checkpoint — it requires the SLAM-LLM inference pipeline plus the two frozen base models:

1. Base LLM: google/gemma-3-4b-it
2. Speech encoder: data2vec-AQC (SPRING_INX Hindi-pretrained checkpoint)

git clone https://github.com/X-LANCE/SLAM-LLM
cd SLAM-LLM

python examples/asr_librispeech/inference_asr_batch.py \
    --config-path "conf" \
    --config-name "prompt_bilingual_ctx.yaml" \
    ++model_config.llm_name=gemma-3-4b-it \
    ++model_config.llm_path=<path_to_gemma-3-4b-it> \
    ++model_config.llm_dim=2560 \
    ++model_config.encoder_name=data2vec_aqc \
    ++model_config.encoder_path=<path_to_data2vec_aqc_checkpoint> \
    ++model_config.encoder_dim=1024 \
    ++model_config.encoder_projector=linear \
    ++model_config.encoder_projector_ds_rate=5 \
    ++train_config.use_peft=true \
    ++train_config.peft_config.peft_method=lora \
    ++train_config.peft_config.r=8 \
    ++train_config.peft_config.lora_alpha=32 \
    ++train_config.peft_config.target_modules=[q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj] \
    ++ckpt_path=<path_to_adapter_model.bin> \
    ++dataset_config.val_data_path=<your_test_jsonl> \
    ++decode_log=<output_prefix>

Input jsonl format (one line per utterance):

{"key": "utt_001", "source": "/path/to/audio.wav", "target": "", "language": "hi-en", "prev_context": ""}

See config.json in this repo for the full architecture/hyperparameter spec.

Prompt Format

<start_of_turn>user
Transcribe speech to {lang} text.{lang_note} If available, use this previous context: {prev_context}<end_of_turn>
<start_of_turn>model

Where {lang} is Hindi, English, or Hindi-English, and {lang_note} adds " This utterance code-switches between Hindi and English." for code-switch utterances.

License

Apache 2.0. Note that the base google/gemma-3-4b-it model is subject to its own Gemma license — this release only contains the LoRA adapter and projector deltas, not the base model weights.

Citation

If you use this model, please cite the SLAM-LLM framework and this release.