zipformer-streaming-robust-sw

Pruned Stateless Zipformer RNN-T Streaming Robust SW

Pruned Stateless Zipformer RNN-T Streaming Robust SW is an automatic speech recognition model trained on the following datasets:

• ALFFA Swahili
• FLEURS Swahili
• Common Voice 16.1 Swahili

Instead of being trained to predict sequences of words, this model was trained to predict sequence of phonemes, e.g. ["w", "ɑ", "ʃ", "i", "ɑ"]. Therefore, the model's vocabulary contains the different IPA phonemes found in gruut.

This model was trained using icefall framework. All training was done on a Scaleway RENDER-S VM with a NVIDIA H100 GPU. All necessary scripts used for training could be found in the Files and versions tab, as well as the Training metrics logged via Tensorboard.

Evaluation Results

Simulated Streaming

for m in greedy_search fast_beam_search modified_beam_search; do
  ./zipformer/decode.py \
    --epoch 40 \
    --avg 7 \
    --causal 1 \
    --chunk-size 32 \
    --left-context-frames 128 \
    --exp-dir zipformer/exp-causal \
    --use-transducer True --use-ctc True \
    --decoding-method $m
done

./zipformer/ctc_decode.py \
    --epoch 40 \
    --avg 7 \
    --causal 1 \
    --chunk-size 32 \
    --left-context-frames 128 \
    --exp-dir zipformer/exp-causal \
    --decoding-method ctc-decoding \
    --use-transducer True --use-ctc True

The model achieves the following phoneme error rates on the different test sets:

Chunk-wise Streaming

for m in greedy_search fast_beam_search modified_beam_search; do
  ./zipformer/streaming_decode.py \
    --epoch 40 \
    --avg 7 \
    --causal 1 \
    --chunk-size 32 \
    --left-context-frames 128 \
    --exp-dir zipformer/exp-causal \
    --use-transducer True --use-ctc True \
    --decoding-method $m \
    --num-decode-streams 1000
done

The model achieves the following phoneme error rates on the different test sets:

Usage

Download Pre-trained Model

cd egs/bookbot_sw/ASR
mkdir tmp
cd tmp
git lfs install
git clone https://huggingface.co/bookbot/zipformer-streaming-robust-sw/

Inference

To decode with greedy search, run:

./zipformer/jit_pretrained_streaming.py \
  --nn-model-filename ./tmp/zipformer-streaming-robust-sw/exp-causal/jit_script_chunk_32_left_128.pt \
  --tokens ./tmp/zipformer-streaming-robust-sw/data/lang_phone/tokens.txt \
  ./tmp/zipformer-streaming-robust-sw/test_waves/sample1.wav

<details> <summary>Decoding Output</summary>

2024-03-07 11:07:41,231 INFO [jit_pretrained_streaming.py:184] device: cuda:0
2024-03-07 11:07:41,865 INFO [jit_pretrained_streaming.py:197] Constructing Fbank computer
2024-03-07 11:07:41,866 INFO [jit_pretrained_streaming.py:200] Reading sound files: ./tmp/zipformer-streaming-robust-sw/test_waves/sample1.wav
2024-03-07 11:07:41,866 INFO [jit_pretrained_streaming.py:205] torch.Size([125568])
2024-03-07 11:07:41,866 INFO [jit_pretrained_streaming.py:207] Decoding started
2024-03-07 11:07:41,866 INFO [jit_pretrained_streaming.py:212] chunk_length: 64
2024-03-07 11:07:41,866 INFO [jit_pretrained_streaming.py:213] T: 77
2024-03-07 11:07:41,876 INFO [jit_pretrained_streaming.py:229] 0/130368
2024-03-07 11:07:41,877 INFO [jit_pretrained_streaming.py:229] 4000/130368
2024-03-07 11:07:41,878 INFO [jit_pretrained_streaming.py:229] 8000/130368
2024-03-07 11:07:41,879 INFO [jit_pretrained_streaming.py:229] 12000/130368
2024-03-07 11:07:42,103 INFO [jit_pretrained_streaming.py:229] 16000/130368
2024-03-07 11:07:42,104 INFO [jit_pretrained_streaming.py:229] 20000/130368
2024-03-07 11:07:42,126 INFO [jit_pretrained_streaming.py:229] 24000/130368
2024-03-07 11:07:42,127 INFO [jit_pretrained_streaming.py:229] 28000/130368
2024-03-07 11:07:42,128 INFO [jit_pretrained_streaming.py:229] 32000/130368
2024-03-07 11:07:42,151 INFO [jit_pretrained_streaming.py:229] 36000/130368
2024-03-07 11:07:42,152 INFO [jit_pretrained_streaming.py:229] 40000/130368
2024-03-07 11:07:42,175 INFO [jit_pretrained_streaming.py:229] 44000/130368
2024-03-07 11:07:42,176 INFO [jit_pretrained_streaming.py:229] 48000/130368
2024-03-07 11:07:42,177 INFO [jit_pretrained_streaming.py:229] 52000/130368
2024-03-07 11:07:42,200 INFO [jit_pretrained_streaming.py:229] 56000/130368
2024-03-07 11:07:42,201 INFO [jit_pretrained_streaming.py:229] 60000/130368
2024-03-07 11:07:42,224 INFO [jit_pretrained_streaming.py:229] 64000/130368
2024-03-07 11:07:42,226 INFO [jit_pretrained_streaming.py:229] 68000/130368
2024-03-07 11:07:42,226 INFO [jit_pretrained_streaming.py:229] 72000/130368
2024-03-07 11:07:42,250 INFO [jit_pretrained_streaming.py:229] 76000/130368
2024-03-07 11:07:42,251 INFO [jit_pretrained_streaming.py:229] 80000/130368
2024-03-07 11:07:42,252 INFO [jit_pretrained_streaming.py:229] 84000/130368
2024-03-07 11:07:42,275 INFO [jit_pretrained_streaming.py:229] 88000/130368
2024-03-07 11:07:42,276 INFO [jit_pretrained_streaming.py:229] 92000/130368
2024-03-07 11:07:42,299 INFO [jit_pretrained_streaming.py:229] 96000/130368
2024-03-07 11:07:42,300 INFO [jit_pretrained_streaming.py:229] 100000/130368
2024-03-07 11:07:42,301 INFO [jit_pretrained_streaming.py:229] 104000/130368
2024-03-07 11:07:42,325 INFO [jit_pretrained_streaming.py:229] 108000/130368
2024-03-07 11:07:42,326 INFO [jit_pretrained_streaming.py:229] 112000/130368
2024-03-07 11:07:42,349 INFO [jit_pretrained_streaming.py:229] 116000/130368
2024-03-07 11:07:42,350 INFO [jit_pretrained_streaming.py:229] 120000/130368
2024-03-07 11:07:42,351 INFO [jit_pretrained_streaming.py:229] 124000/130368
2024-03-07 11:07:42,373 INFO [jit_pretrained_streaming.py:229] 128000/130368
2024-03-07 11:07:42,374 INFO [jit_pretrained_streaming.py:259] ./tmp/zipformer-streaming-robust-sw/test_waves/sample1.wav
2024-03-07 11:07:42,374 INFO [jit_pretrained_streaming.py:260] ʃiɑ|ɑᵐɓɑɔ|wɑnɑiʃi|hɑsɑ|kɑtikɑ|ɛnɛɔ|lɑ|mɑʃɑɾiki|kɑtikɑ|ufɑlmɛ|huɔ|wɛnjɛ|utɑʄiɾi|wɑ|mɑfutɑ
2024-03-07 11:07:42,374 INFO [jit_pretrained_streaming.py:262] Decoding Done

</details>

Training procedure

Install icefall

git clone https://github.com/bookbot-hive/icefall
cd icefall
export PYTHONPATH=`pwd`:$PYTHONPATH

Prepare Data

cd egs/bookbot_sw/ASR
./prepare.sh

Train

export CUDA_VISIBLE_DEVICES="0"
./zipformer/train.py \
  --num-epochs 40 \
  --use-fp16 1 \
  --exp-dir zipformer/exp-causal \
  --causal 1 \
  --max-duration 800 \
  --use-transducer True --use-ctc True

Frameworks

• k2
• icefall
• lhotse