hindi-hinglish-zipformer-ctc-streaming

Hindi/Hinglish Streaming ASR — Zipformer-M CTC

A from-scratch streaming ASR model for Hindi + code-mixed Hinglish. Architecture: Zipformer-M encoder (~65M) + pure CTC head, causal/chunked for streaming, character-level (Unicode codepoint) vocab. Trained with icefall.

Two releases:

• model_p3_epoch15_avg8.pt — CURRENT BEST (Phase 3): scaled up on ~337h.
• model_p1_epoch30_avg8.pt — Phase 1 baseline (~89h).

Results (CTC greedy, no LM; checkpoint-averaged)

Architecture

Zipformer2 "M" (~64.6M): num_encoder_layers=2,2,3,4,3,2, encoder_dim=192,256,384,512,384,256, downsampling_factor=1,2,4,8,4,2, causal, chunk_size=16,32,64,-1, left_context_frames=64,128,256,-1. Pure CTC head, 80-dim fbank @16kHz, subsampling 4 (~25 Hz). Vocab 103 tokens (<blk>,<sos/eos>,<unk>,▁=word-boundary, 63 Devanagari + 26 a-z + 10 digits).

Training data

• Phase 1 (base, from scratch): IndicVoices-hi 60h + keywords 15.4h + Svarah 8.4h + numbers 5.9h (~89h).
• Phase 3 (scale-up, warm-start from P1): IndicVoices-hi 307h + keywords + Svarah + numbers (~337h base → ~1017h with ×3 speed perturbation). IndicVoices text uses unsanitized_normalized (code-switch word [English] → English, tags stripped); all text NFC-normalized, punctuation/foreign scripts dropped, Latin lowercased.

Training

icefall zipformer recipe (char-CTC), PyTorch 2.4.1+cu124 / k2 / lhotse, bf16 (fp16 was unstable on a single L4), ScaledAdam + Eden. P1: from scratch, base_lr 0.045, 30 ep. P3: warm-start (--init-ckpt), base_lr 0.02, 15 ep, non-finite-loss batches skipped. Single NVIDIA L4.

Files

model_p3_epoch15_avg8.pt / model_p1_epoch30_avg8.pt — averaged weights + arch config + vocab_size. tokens.txt — vocab. scripts/ — full reproduction pipeline.

Usage (decode)

Build the icefall zipformer from the checkpoint's config, load_state_dict(ckpt["model"]), extract 80-dim lhotse fbank, run encoder_embed → encoder → ctc_output → ctc_greedy_search, map ids via tokens.txt, replace ▁→space. See scripts/eval_wer.py and scripts/server_ctc.py.

Limitations

Greedy CTC, no LM (n-gram/RNN-T would improve WER). English weaker than Hindi (English data-limited).