STT-streaming-zipformer-indian-en

STT Streaming Zipformer for Indian English

A streaming ASR model fine-tuned for Indian-accented English, optimized for real-time on-device inference (Android/iOS) via sherpa-onnx.

Performance

Word Error Rate on the Svarah benchmark (9.6 hours, 117 Indian English speakers across 19 states):

| Configuration | WER | Notes |

|---|---|---|

| icefall greedy_search (chunk-32) | 24.01% | Initial baseline |

| icefall greedy + averaged epochs 6-10 | 21.70% | Checkpoint averaging |

| icefall greedy + chunk-64 ⭐ | 20.73% | Best icefall result |

| sherpa-onnx int8 + greedy_search | 27.96% | Production deployment |

| sherpa-onnx float32 + modified_beam_search | 26.90% | Best sherpa-onnx |

For reference, the base streaming Zipformer (LibriSpeech only, before fine-tuning) achieves 41.03% WER on Svarah — fine-tuning reduced errors by 49.5% relative.

Comparison with other systems on Svarah

| System | Params | WER |

|---|---|---|

| Sarvam Saaras V3 | proprietary | 6.37% |

| Whisper Large | 1.55B | 9.1% |

| Whisper Medium | 769M | 11.2% |

| Conformer Large (NVIDIA) | 120M | 14.6% |

| Google Cloud STT en-IN | commercial | 20.7% |

| Azure STT en-US | commercial | 20.9% |

| Azure STT en-IN | commercial | 21.3% |

| This model (icefall) | 66M | 20.73% |

| Data2Vec Large | 313M | 24.5% |

| Wav2Vec2 Large | 317M | 24.9% |

| HuBERT Large | 316M | 25.6% |

| Google Cloud STT en-US | commercial | 30.0% |

| WavLM Large | 300M | 33.7% |

| Base Zipformer (no fine-tune) | 66M | 41.0% |

Model Details

• Architecture: Streaming Zipformer-Transducer (RNN-T)

• Parameters: 66.1 million

• Encoder layers: 2,2,3,4,3,2 (6 stacks)

• Encoder dimensions: 192,256,384,512,384,256

• Causal/Streaming: Yes (suitable for real-time inference)

• Chunk size: 64 frames (640 ms)

• Left context: 256 frames (2.56 s)

• BPE vocab size: 500

• Sample rate: 16000 Hz

• Feature: 80-dim fbank

• Fine-tuned from: Zengwei/icefall-asr-librispeech-streaming-zipformer-2023-05-17

Files

| File | Size | Purpose |

|---|---|---|

| encoder-epoch-10-avg-5-chunk-64-left-256.int8.onnx | 67 MB | Encoder (int8 quantized, recommended) |

| decoder-epoch-10-avg-5-chunk-64-left-256.int8.onnx | 0.5 MB | Decoder (int8 quantized) |

| joiner-epoch-10-avg-5-chunk-64-left-256.int8.onnx | 0.25 MB | Joiner (int8 quantized) |

| encoder-epoch-10-avg-5-chunk-64-left-256.onnx | 249 MB | Encoder (float32) |

| decoder-epoch-10-avg-5-chunk-64-left-256.onnx | 2.0 MB | Decoder (float32) |

| joiner-epoch-10-avg-5-chunk-64-left-256.onnx | 1.0 MB | Joiner (float32) |

| tokens.txt | 5 KB | BPE vocabulary file |

| bpe.model | 245 KB | SentencePiece BPE model |

Total int8: ~68 MB | Total float32: ~252 MB

Inference Speed

Tested on Intel Xeon CPU (4 threads):

• RTF (int8): 0.035 → 28.5x realtime

• RTF (float32): 0.183 → 5.5x realtime

• Latency per chunk: 640 ms

On modern Android phones (Snapdragon 7-series and up), expect 5-10x realtime with int8.

Usage with sherpa-onnx (Python)

import sherpa_onnx

import wave

import numpy as np



recognizer = sherpa_onnx.OnlineRecognizer.from_transducer(

    encoder='encoder-epoch-10-avg-5-chunk-64-left-256.int8.onnx',

    decoder='decoder-epoch-10-avg-5-chunk-64-left-256.int8.onnx',

    joiner='joiner-epoch-10-avg-5-chunk-64-left-256.int8.onnx',

    tokens='tokens.txt',

    num_threads=4,

    sample_rate=16000,

    feature_dim=80,

    decoding_method='greedy_search',

    provider='cpu',

)



with wave.open('audio.wav', 'rb') as f:

    assert f.getframerate() == 16000

    audio = np.frombuffer(f.readframes(f.getnframes()), dtype=np.int16)

    audio = audio.astype(np.float32) / 32768.0



stream = recognizer.create_stream()

stream.accept_waveform(16000, audio)

tail_padding = np.zeros(int(0.5 * 16000), dtype=np.float32)

stream.accept_waveform(16000, tail_padding)

stream.input_finished()



while recognizer.is_ready(stream):

    recognizer.decode_stream(stream)



print(recognizer.get_result(stream))

Usage on Android

This model is designed for sherpa-onnx Android bindings. Drop the int8 ONNX files plus tokens.txt into your app's assets folder and follow the sherpa-onnx Kotlin streaming example.

Training Data

Fine-tuned on a curated subset of NPTEL (National Programme on Technology Enhanced Learning) Indian English lectures:

• Total: 277.4 hours (train) + 14.6 hours (dev)

• Speakers: 198 unique professors (clustered via ECAPA-TDNN embeddings)

• Quality: Mean SNR 34 dB, mean quality score 0.96

• Domains: Engineering, science, mathematics, humanities lectures

• Transcripts: Generated via Whisper Large-v3 (faster-whisper int8_float16)

• Filtering: Quality score >=0.90, SNR >=20 dB, duration 3-15s, max 8 hrs/speaker

Training Setup

• Framework: icefall zipformer recipe

• Hardware: 2x NVIDIA RTX A6000 (48 GB each)

• Distributed: PyTorch DDP

• Precision: FP16 mixed-precision

• Optimizer: ScaledAdam

• Base learning rate: 0.0045

• Batch size: 600 seconds per GPU

• Epochs: 20 (best validation at epoch 8)

• Total training time: ~12 hours

Limitations

• Conversational speech: Trained on academic lectures; performance degrades on casual/spontaneous speech (banking, customer service, etc.)

• Numbers and digits: Training transcripts had numbers stripped — model outputs ⁇ for digits. Use post-processing or fine-tune further on data with numerics.

• Code-mixed speech: Limited support for Hindi/Indic words mixed within English (e.g., "Yojna", "Yaar", "Lakh")

• Streaming gap: sherpa-onnx production WER (~27%) is ~6% higher than icefall batched eval (20.73%) due to streaming behavior differences

Not Suitable For

• High-stakes transcription requiring <5% WER

• Conversational/casual speech without further fine-tuning

• Code-mixed Hindi-English content

• Audio with heavy background noise (model was trained on clean lecture audio)

Citation

If you use this model, please consider citing the underlying frameworks:

@inproceedings{Yao2023Zipformer,

  title={Zipformer: A faster and better encoder for automatic speech recognition},

  author={Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel},

  booktitle={ICLR},

  year={2024}

}



@misc{javed2023svarah,

  title={Svarah: Evaluating English ASR Systems on Indian Accents},

  author={Tahir Javed and Sakshi Joshi and Vignesh Nagarajan and Sai Sundaresan and Janki Nawale and Abhigyan Raman and Kaushal Bhogale and Pratyush Kumar and Mitesh M. Khapra},

  year={2023}

}

Acknowledgements

• AI4Bharat (IIT Madras) for the Svarah benchmark

• k2-fsa team for icefall, k2, lhotse, sherpa-onnx

• Zengwei Yao et al. for the streaming Zipformer architecture

• NPTEL for the lecture audio used in fine-tuning