sherpa-kws

Model Card for Sherpa-ONNX KWS Zipformer zh-en 3M (IB-Robot)

A streaming Zipformer2 transducer keyword spotter (3M parameters, Chinese + English) packaged for the IB-Robot framework and compiled for Ascend 310P1. This is the always-on stage: it runs on the idle robot and decides when to wake the rest of the pipeline (VAD → STT → dialogue → TTS).

Keywords are configured at run time from a text file of token sequences — no retraining is needed to change the wake word.

Deployments

Shared contract: tensor_model/zipformer2_transducer_kws/keyword_spotting (host.kws.features [1,45,80] -> host.kws.logit [1,263]), preprocessing kaldi-fbank80-16khz-povey-snipedgesfalse-v1, output keyword-label-on-trigger-v1, logical revision sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64.

Both deployments are stateful: the encoder carries 38 cache tensors across chunks (state_scope: stream, execution_structure: iterative, cancellation_granularity: checkpoint). The manifest declares them as a single state bank encoder.bank, linking each *_in to the matching *_out.

The contract is orchestration_visibility: session, not executor: the caller owns the loop. Each call runs one encoder → decoder → joiner pass over a single chunk; advancing the caches, shifting the feature buffer and running the beam search are the session's job — see assets/reference_impl/live_kws.py.

Tensor contract

encoder — 1 feature input + 38 state inputs, 1 output + 38 state outputs:

x            float32 [1, 45, 80]    45 fbank frames, 80 mel bins
encoder_out  float32 [1,  8, 320]   8 output frames per chunk

The chunk is 45 frames in / 8 frames out and advances by 32 frames (decode_chunk_len = 32), i.e. each step consumes 320 ms of new audio while seeing 130 ms of right context. This is the chunk-16-left-64 streaming variant.

State tensors (all zeros at stream start, fed back verbatim), 6 encoder layers with left_context = [64,32,16,8,16,32], key_dim = [128,128,128,256,128,128], value_dim = [48,48,48,96,48,48]:

cached_key_{i}          [left_context[i], 1, key_dim[i]]
cached_nonlin_attn_{i}  [1, 1, left_context[i], 96]
cached_val1_{i}         [left_context[i], 1, value_dim[i]]
cached_val2_{i}         [left_context[i], 1, value_dim[i]]
cached_conv1_{i}        [1, 128, 7]
cached_conv2_{i}        [1, 128, 7]
embed_states            [1, 128, 3, 19]
processed_lens          int64 [1]        frames consumed so far, +32 per chunk

decoder — y int64 [1, 2] (context size 2) → decoder_out float32 [1, 320].

joiner — encoder_out [1, 320] + decoder_out [1, 320] → logit [1, 263].

Feature extraction

Kaldi-compatible fbank: 16 kHz mono, 80 mel bins, 25 ms window / 10 ms shift, povey window, snip_edges = false, no dithering, no CMVN.

snip_edges = false defines frame centres over the whole signal with mirrored padding, so features cannot be computed per audio block independently — doing so produces wrong frames at every block boundary. The reference implementation keeps a rolling PCM buffer, recomputes fbank over it, and emits only newly-stable frames, dropping the last 3 (their right context has not arrived yet). See assets/reference_impl/live_kws.py.

Repository Structure

• inference_manifest.json — deployment routing (schema v3)
• assets/adapter.json — adapter identity (zipformer2_transducer_kws / keyword_spotting)
• assets/tokens.txt — 263-entry token table, English ARPAbet phones plus pinyin syllables with tone marks (needed to encode keywords)
• assets/keywords.txt, assets/keywords_sensitive.txt — example keyword sets (default and higher-sensitivity thresholds)
• assets/en.phone — English lexicon required to encode new keywords
• assets/onnx/*.onnx — upstream ONNX models (the torch_cpu deployment and the ATC compilation source)
• assets/pretrained-epoch-13-avg-2.pt — the upstream PyTorch training checkpoint the ONNX graphs were exported from (13 MB, averaged weights)
• assets/reference_impl/np_fbank.py — dependency-free numpy fbank
• assets/reference_impl/kws_decode.py — context graph + transducer keyword beam search (max_active_paths=4, num_trailing_blanks=1, unk_id=2)
• assets/reference_impl/live_kws.py — end-to-end streaming driver
• artifacts/ascend_310p/*.om
• artifacts/ascend_310b/*.om — 310B compiled trio

Source Model

The origin is the icefall training repository pkufool/icefall-kws-zipformer-zh-en-3M-2025-12-20 on ModelScope (branch master, declared license Apache-2.0), checkpoint epoch-13-avg-2, streaming variant chunk-16-left-64, logical revision sherpa-onnx-kws-zipformer-zh-en-3M@2025-12-20-chunk16-left64.

The widely-cited download is the sherpa-onnx GitHub release tarball, but that tarball redistributes this repository's onnx/ directory — all three ONNX graphs we ship are byte-identical to the files there, which is how the origin was identified rather than assumed:

Only the encoder distinguishes the streaming variants — decoder and joiner are the same bytes for chunk-8-left-64, chunk-16-left-64 and chunk-16-left-128 in that repository. The variant is therefore pinned by the encoder hash above, and independently confirmed by matching the ONNX metadata (T=45, decode_chunk_len=32) against the streaming driver's constants rather than trusting a file name. chunk-16 is the 320 ms-latency variant; the 160 ms chunk-8 variant exists upstream and is not used here.

The training checkpoint is shipped (assets/pretrained-epoch-13-avg-2.pt, the averaged weights the ONNX export was produced from), matching the convention in the rest of this organisation, where the upstream checkpoint lives under assets/. The repository also holds checkpoint/epoch-13.pt (51,959,636 bytes, full training state including optimizer); that is not needed for inference or re-export and is not shipped.

assets/en.phone is not in the ModelScope repository — it exists only in the sherpa-onnx release tarball, so that remains the source for it:

wget https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models/\
sherpa-onnx-kws-zipformer-zh-en-3M-2025-12-20.tar.bz2
# 32,885,699 bytes
# sha256 68447f4fbc67e70eee3a93961f36e81e98f47aef73ce7e7ca00885c6cd3616a6

The five files taken from that archive (three ONNX, tokens.txt, en.phone) were each verified byte-identical against a fresh download as well, so the two sources agree.

The ONNX export is not byte-reproducible from the checkpoint here: the icefall export script is not part of this bundle. What is pinned is the exact checkpoint, the exact exported graphs, and the fact that the two come from the same upstream commit.

Compiled on the board with ATC (--soc_version=Ascend310P1):

The upstream repository declares Apache-2.0 but publishes no training-corpus description or dataset licence for this release.

Validation

Measured on a real Ascend 310P1 with the streaming driver, 8.5 s of 16 kHz audio containing two wake-word utterances:

chunks              26
mean compute        91 ms / chunk   (OM inference + keyword decode)
audio per chunk    320 ms
RTF                0.29
detections         2.37 s, 5.25 s   (both expected utterances, no false alarms)
backlog            30–50 ms         (stable — the pipeline keeps up)

RTF 0.29 leaves ~3× headroom, and the backlog stays flat rather than growing, which is the real test for a streaming model: it is consuming audio at least as fast as it arrives.

OM vs ONNX, tensor level

Continuous streaming comparison on the same 310P1: both backends run the full 45-frame-in / 32-frame-shift loop over the upstream test_wavs, each keeping its own 38 cache tensors, so any error accumulates exactly as it would in production. ONNX Runtime 1.23.2 CPU fp32 is the reference.

decoder cosine 0.99999991, joiner cosine 0.99999992, joiner argmax agreement 1.00 — the OM and ONNX joiners pick the same token every time. processed_lens (the int64 frame counter carried through the state) is exactly equal on every chunk of every file.

The state does not drift. The worst state cosine is reached mid-stream (chunk 1, 6 and 14 respectively) and the value recovers — after 51 chunks of continuous streaming, 16.7 s of audio, en_1 is back at 0.9991. The error is per-chunk numerical noise, not accumulation, which is the failure mode that would actually matter for a model whose caches never reset.

OM vs ONNX, end to end

The number that decides whether the OM is usable is not a cosine, it is whether detections change. Same audio, same decoder, same context graph — only the backend differs:

9 files, 10 detections, zero disagreement — same keyword, same chunk. The three files with no listed keyword produce no hits on either backend, so this covers false alarms as well as misses. The detections also match the upstream ground truth (test_wavs/en_trans.txt, keywords_raw.txt), so both backends are right, not merely consistent.

Usage

from inference_manifest import load_inference_manifest

validated = load_inference_manifest("models/sherpa-kws", "ascend_310p")

Per chunk: run the encoder on 45 fbank frames plus the carried state, feed the 8 output frames to the transducer keyword decoder, keep the returned state, and slide the feature buffer forward by 32 frames.

A standalone run, reading raw 16 kHz mono s16le PCM from stdin:

parec --device=<src> --rate=16000 --channels=1 --format=s16le \
  | python3 live_kws.py --model-dir . --tokens tokens.txt --keywords keywords.txt

Defining your own keywords

The keyword file holds token sequences, not plain text, so a new wake word has to be encoded first. This model is bilingual and needs both the pinyin and the English lexicon:

# keywords_raw.txt: one keyword per line, spaces replaced by underscores,
# and the original text after an "@", e.g.:  n i3 h ao3 @你好
sherpa-onnx-cli text2token \
  --tokens tokens.txt \
  --tokens-type phone+ppinyin \
  --lexicon en.phone \
  keywords_raw.txt keywords.txt

Each line then accepts optional per-keyword overrides: :score boosts the context-graph bonus, #threshold sets the detection threshold for that keyword alone. keywords_sensitive.txt is the same keyword set at a lower threshold — more hits, more false alarms. Tune per keyword rather than globally: a short wake word needs a higher threshold than a long one.

No retraining is involved; changing the wake word is a text-file change.

License

Code and packaging: Apache-2.0. Model weights follow the upstream sherpa-onnx / k2-fsa license (Apache-2.0).

Citation

@article{zipformer2023,
  title   = {Zipformer: A faster and better encoder for automatic speech recognition},
  author  = {Yao, Zengwei and Guo, Liyong and Yang, Xiaoyu and Kang, Wei and Kuang, Fangjun and Yang, Yifan and Jin, Zengrui and Lin, Long and Povey, Daniel},
  journal = {arXiv preprint arXiv:2310.11230},
  year    = {2023}
}
@software{sherpa_onnx,
  title   = {sherpa-onnx: Real-time speech recognition and keyword spotting},
  author  = {{Next-gen Kaldi (k2-fsa)}},
  url     = {https://github.com/k2-fsa/sherpa-onnx},
  license = {Apache-2.0}
}
@software{ib_robot,
  title   = {IB-Robot: Intelligence Boom Robot},
  url     = {https://atomgit.com/openeuler/IB_Robot},
  license = {Apache-2.0}
}