zipformer-streaming-robust-es-v1

Pruned Stateless Zipformer RNN-T Streaming Robust ES v1

zipformer-streaming-robust-es-v1 is Bookbot's causal Spanish pruned-stateless Zipformer RNN-T phoneme recognizer. It emits atomic, whitespace-interpretable IPA phone tokens rather than words. The canonical inference state is the exact cumulative averaged-model difference over epochs 76–80; its 639 tensors were checked against the production Icefall averaging path before the source checkpoints were pruned.

License and permitted use

The model weights are published under CC BY-NC-SA 3.0 because training included the TinyVox Spanish subset, which is derived from TalkBank/PhonBank data under that license and is restricted to non-commercial use. Do not use these weights in a commercial product or service. Follow the TalkBank Ground Rules, retain attribution, and share adaptations under compatible terms.

The included Icefall inference script retains its upstream Apache-2.0 source-code license. Common Voice and SLR72 retain their own source licenses. TinyVox audio, metadata, and utterance-level recognition output are not included in this repository or the companion public datasets.

Training data

• Common Voice 23.0 Spanish, CC0-1.0.
• OpenSLR72 Colombian Spanish, CC BY-SA 4.0.
• TinyVox Spanish, CC BY-NC-SA 3.0; source-hosted only.

Versioned public training datasets with the exact v1 schema are:

• bookbot/common-voice-23-0-es-ipa-v1
• bookbot/slr72-es-ipa-v1

Each row contains only id, audio, and the exact canonical ipa_transcript consumed by this model.

Output inventory

data/lang_phone/tokens.txt is the authoritative symbol table. It contains 28 non-disambiguation entries, including <eps> and <UNK>, plus #0 for graph disambiguation. Phones are atomic tokens; the composite affricate t͡ʃ is one token. The vocabulary differs from v0's older BabyGruut-era inventory and is not numerically compatible with v0 checkpoints.

Streaming configuration

• Architecture: pruned stateless Zipformer RNN-T
• Causal streaming: yes
• Chunk size: 16 encoder frames
• Left context: 128 frames
• Encoder layers: 2,2,2,2,2,2
• Feed-forward dimensions: 512,768,768,768,768,768
• Encoder dimensions: 192,256,256,256,256,256
• Encoder unmasked dimensions: 192,192,192,192,192,192
• Decoder and joiner dimensions: 512
• Training hardware: 2 NVIDIA RTX 4090 GPUs

Published artifacts

• epoch-80.pt: full resumable final training state.
• evaluated-avg-76-80.pt: canonical inference checkpoint used for every reported result.
• jit_script_chunk_16_left_128.pt: TorchScript export from that exact canonical checkpoint.
• data/lang_phone/: token, lexicon, and graph artifacts.
• jit_pretrained_streaming.py: streaming greedy inference example.
• test_waves/common_voice_es_19696062.wav: CC0 Common Voice 23.0 Spanish sample, resampled to 16 kHz.
• streaming/evaluation-common-voice-23-0-es-slr72.json: machine-readable corpus provenance and results.
• model-config.json and SHA256SUMS: reproducibility metadata.
• tensorboard/ and log/: training metrics and logs.

The sherpa-onnx ORT conversion is published separately at bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1.

Setup and usage

Install Icefall and its dependencies following the Icefall installation guide. Bookbot uses its Icefall fork:

git clone https://github.com/bookbot-hive/icefall
cd icefall/egs/bookbot_es/ASR

Download and run the pre-trained model

Clone this repository and run the included sample:

git lfs install
mkdir -p tmp
git clone \
  https://huggingface.co/bookbot/zipformer-streaming-robust-es-v1 \
  tmp/zipformer-streaming-robust-es-v1

python tmp/zipformer-streaming-robust-es-v1/jit_pretrained_streaming.py \
  --nn-model-filename tmp/zipformer-streaming-robust-es-v1/jit_script_chunk_16_left_128.pt \
  --tokens tmp/zipformer-streaming-robust-es-v1/data/lang_phone/tokens.txt \
  tmp/zipformer-streaming-robust-es-v1/test_waves/common_voice_es_19696062.wav

Observed greedy output for the included sample:

xeneɾalmentelosimmigɾantessesitwaɾonenlapeɾifeɾjadelassjudades

The underlying token sequence exactly matches the production checkpoint's stored greedy hypothesis:

x e n e ɾ a l m e n t e l o s i m m i g ɾ a n t e s s e s i t w a ɾ o n e n l a p e ɾ i f e ɾ j a d e l a s s j u d a d e s

Input audio must be mono 16 kHz WAV or another format supported by torchaudio with a 16 kHz sample rate.

Evaluation

All values are phoneme error rate (PER). Icefall's generic error writer and historical wer-summary-* filenames use the label WER, but the aligned units are whitespace-separated IPA phones.

Evaluation uses the exact epochs 76–80 averaged model, causal chunk size 16, left context 128, and 1,000 decode streams.

Use modified beam search with num_active_paths=4 for this model. The reference fast-beam configuration collapses through excessive blank/deletion bias. A blank penalty selected only on TinyVox validation rescues that collapse but remains worse than modified beam on every reported corpus.

Evaluation limits

The recipe reuses the Common Voice and SLR72 official test splits for validation, so those two results are validation-overlapped rather than independent holdouts. TinyVox test is the independently held-out result; it was evaluated after the fast-beam setting had been selected on TinyVox validation. Common Voice dominates any combined aggregate, so per-corpus values are preferred.

Checkpoint provenance

• Best recorded validation loss: 0.1675387544560873 at epoch 75.
• Training completed the configured 80 epochs; this completed run did not early-stop.
• Canonical averaged inference range: epochs 76–80.
• epoch-80.pt SHA-256: e6704a4c639f94e7361fb07d6fac698d3ea0c67d8f3eecef4ad6c0423013b646.
• evaluated-avg-76-80.pt SHA-256: d5cde97d82c214da39c3d9decffa84eaf6b15869c7b6521d2f184a2e59ad30e5.

See SHA256SUMS for every packaged binary and model-config.json for exact export settings.

Training procedure

The historical run used Python 3.12, PyTorch 2.4.0+cu124, k2 1.24.4, Lhotse 1.31.1, and two NVIDIA RTX 4090 GPUs. The retained checkpoint records the clean Icefall branch feature/spanish-phonemization-retrain at 4b746f3b. That recipe revision is the authority for the historical run; the current launcher may contain later checkpoint-retention and early-stopping policy that did not produce these weights.

Install Icefall

git clone https://github.com/bookbot-hive/icefall
cd icefall
export PYTHONPATH="$PWD:${PYTHONPATH:-}"
cd egs/bookbot_es/ASR

Install Icefall and its k2/Lhotse dependencies using the official installation guide. The exact 4b746f3b recipe revision must be available in your checkout for historical data preparation. At publication time that feature revision was recorded by the checkpoint but was not reachable from the public fork, so this is an explicit source-reproducibility limit rather than a claim that current master is identical.

Prepare data

The two public companion datasets expose the accepted Common Voice and SLR72 rows, but exact training additionally requires:

• pinned full-source snapshots obtained from Mozilla Data Collective and OpenSLR72, identified by historical preparation revisions 591a99e9c5c9e308231746c7baa03eadfe670483 and e7c9924b964eecbdcba1181add860367d3c7ca3a, respectively;
• an existing verified Common Voice/SLR72 Lhotse fbank store;
• locally obtained TinyVox Spanish data and the licensed alignment lexicon.

Bookbot no longer hosts those full historical source snapshots. Exact preparation therefore requires preserved local copies matching the recorded revision and manifest hashes. The versioned IPA datasets above contain only accepted final cuts and cannot replace the larger full-source inputs in this historical preparation command.

TinyVox is not redistributed here. Obtain it from TalkBank and comply with CC BY-NC-SA 3.0 and the TalkBank Ground Rules. With those prerequisites, the historical preparation stages are:

python local/prepare_spanish_retrain.py \
  --existing-fbank-dir /path/to/existing/fbank \
  --output-data-dir data \
  --common-voice-source /path/to/common_voice_23_0_es \
  --slr72-source /path/to/slr72_dataset \
  --tinyvox-root /path/to/tinyvox_spanish \
  --tinyvox-alignment-lexicon local/tinyvox_alignment_lexicon.tsv

python local/compute_fbank_tinyvox.py

python local/prepare_lexicon.py \
  --tokens-file es-tokens.txt \
  --lang-dir data/lang_phone

python local/prepare_lang.py \
  --tokens-file es-tokens.txt \
  --lang-dir data/lang_phone

python local/validate_spanish_retrain.py \
  --fbank-dir data/fbank \
  --reports-dir data/reports \
  --tokens-file es-tokens.txt \
  --lang-tokens data/lang_phone/tokens.txt

Preparation is fail-closed: pinned source revisions and manifests, full audio joins, the fixed phone inventory, and TinyVox split isolation must validate before training starts.

Train

This is the effective historical command recorded by epoch-80.pt. The checkpoint confirms keep_last_k=30, seed 42, causal training chunk choices 16,32,64,-1, left-context choices 64,128,256,-1, MUSAN mixing, and SpecAugment. No early stopping was active in the run that produced v1.

export CUDA_VISIBLE_DEVICES="0,1"
export OMP_NUM_THREADS=1

python zipformer/train.py \
  --world-size 2 \
  --num-epochs 80 \
  --start-epoch 1 \
  --exp-dir zipformer/exp-spanish-atomic-phones \
  --manifest-dir data/fbank \
  --lang-dir data/lang_phone \
  --causal 1 \
  --chunk-size "16,32,64,-1" \
  --left-context-frames "64,128,256,-1" \
  --num-encoder-layers 2,2,2,2,2,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --encoder-dim 192,256,256,256,256,256 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --max-duration 1000 \
  --base-lr 0.04 \
  --keep-last-k 30 \
  --seed 42 \
  --enable-musan true \
  --enable-spec-aug true \
  --use-transducer true \
  --use-fp16 1

The run completed epoch 80. evaluated-avg-76-80.pt is the materialized Icefall cumulative averaged-model difference for epochs 76–80, not a renamed single-epoch checkpoint.

Exporting to ONNX

The streaming exporter emits FP32 and dynamic-INT8 encoder, decoder, and joiner ONNX files. Starting in icefall/egs/bookbot_es/ASR, expose the exact materialized average through a scratch numeric checkpoint name:

native=tmp/zipformer-streaming-robust-es-v1
scratch="$(mktemp -d)"
ln -s "$(realpath "$native/evaluated-avg-76-80.pt")" \
  "$scratch/epoch-999.pt"

python zipformer/export-onnx-streaming.py \
  --tokens "$native/data/lang_phone/tokens.txt" \
  --epoch 999 \
  --avg 1 \
  --use-averaged-model false \
  --exp-dir "$scratch" \
  --num-encoder-layers 2,2,2,2,2,2 \
  --downsampling-factor 1,2,4,8,4,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --num-heads 4,4,4,8,4,4 \
  --encoder-dim 192,256,256,256,256,256 \
  --query-head-dim 32 \
  --value-head-dim 12 \
  --pos-head-dim 4 \
  --pos-dim 48 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --cnn-module-kernel 31,31,15,15,15,31 \
  --decoder-dim 512 \
  --joiner-dim 512 \
  --context-size 2 \
  --causal true \
  --chunk-size 16 \
  --left-context-frames 128 \
  --use-transducer true

for component in encoder decoder joiner; do
  mv "$scratch/${component}-epoch-999-avg-1-chunk-16-left-128.onnx" \
    "$scratch/${component}-evaluated-avg-76-80-chunk-16-left-128.onnx"
  mv "$scratch/${component}-epoch-999-avg-1-chunk-16-left-128.int8.onnx" \
    "$scratch/${component}-evaluated-avg-76-80-chunk-16-left-128.int8.onnx"
done

The exporter uses ONNX opset 13 and onnxruntime.quantization.quantize_dynamic with QInt8 weights. It quantizes encoder MatMul, decoder MatMul/Gather, and joiner MatMul operations.

Converting ONNX to ORT

The published package used ONNX Runtime 1.23.2 with fixed optimization:

(
  cd "$scratch"
  python -m onnxruntime.tools.convert_onnx_models_to_ort \
    --optimization_style=Fixed \
    .
)

This produces six .ort files plus required_operators.config. Verify the ORT files before publication, then remove the intermediate ONNX files and the scratch checkpoint alias:

python -c 'import glob, sys, onnxruntime as ort; [ort.InferenceSession(p, providers=["CPUExecutionProvider"]) for p in glob.glob(sys.argv[1] + "/*.ort")]' "$scratch"
rm -f "$scratch"/*.onnx "$scratch"/epoch-999.pt

The final, validated ORT artifacts are published in bookbot/sherpa-onnx-ort-zipformer-streaming-robust-es-v1. export-config.json in that repository records the architecture, quantization operators, conversion settings, and output names.

Exporting the TorchScript model

The materialized average can be exposed to Icefall's numeric checkpoint loader through a temporary alias:

ln -s "$PWD/evaluated-avg-76-80.pt" /tmp/bookbot-v1-export/epoch-999.pt

python zipformer/export.py \
  --tokens data/lang_phone/tokens.txt \
  --use-averaged-model false \
  --epoch 999 --avg 1 \
  --exp-dir /tmp/bookbot-v1-export \
  --jit true --causal true \
  --chunk-size 16 --left-context-frames 128 \
  --num-encoder-layers 2,2,2,2,2,2 \
  --feedforward-dim 512,768,768,768,768,768 \
  --encoder-dim 192,256,256,256,256,256 \
  --encoder-unmasked-dim 192,192,192,192,192,192 \
  --use-transducer true

Remove the temporary alias after export. It is not a third retained checkpoint.

Frameworks

• Icefall
• k2
• Lhotse
• PyTorch

Citations

For Common Voice, OpenSLR72, TinyVox, and TalkBank, follow the citation requirements on the linked source pages. In particular, SLR72 derives from Guevara-Rukoz et al., Crowdsourcing Latin American Spanish for Low-Resource Text-to-Speech (LREC 2020), and TinyVox is described by Charlot et al. (2025) and Lavechin, Bergelson, and Levy (2026).