rekody-streaming-en-0.6b-160ms-int8

rekody-streaming-en-0.6b-160ms-int8

An int8 ONNX conversion of NVIDIA's Nemotron speech streaming 0.6B model (the March 2026 checkpoint), exported at the 160 ms cache-aware latency profile and packaged for on-device streaming dictation. This is the model behind Rekody, the private, local-first voice dictation tool for macOS.

• Cache-aware streaming ASR: transcribes while you speak, in 160 ms chunks, with encoder cache carried across chunks. No audio leaves the machine.
• Runs on the ONNX Runtime CPU execution provider. The files load directly with onnxruntime or parakeet-rs.
• Four files: encoder.onnx (int8, 880 MB), decoder_joint.onnx (int8, 11 MB), tokenizer.model (SentencePiece), and config.json.

What "160 ms" means, and why it is a separate repository

The checkpoint supports several cache-aware right-context settings. The attention right context R fixes how many encoder frames are produced per call, and each frame is 80 ms:

The profile is baked into the exported graph, so this artifact is not a drop-in replacement for the 560 ms one. Feeding this encoder a 560 ms chunk (65 mel frames) returns only the first 2 output frames and silently discards five sevenths of the audio — no error is raised. That is why the two profiles live in separate repositories rather than as revisions of one: a consumer pinned to the 560 ms bytes keeps getting exactly those bytes.

Consumers that read the profile from the file do the right thing automatically. The encoder's ONNX metadata_props carry:

These are the key names and decimal-string format that parakeet-rs >= 0.3.7 reads from the encoder session, defaulting to 7 and 9 when absent. Mel frames per encoder call is chunk_size_output_frames * 8 + pre_encode_cache = 25, i.e. 2560 audio samples of new audio per chunk at 16 kHz.

Accuracy at this profile

NVIDIA publishes word error rate by chunk size for this checkpoint across the eight Open ASR Leaderboard sets. Reproduced here from the base model card as upstream data, not as a Rekody measurement:

The +0.60 pp average gap is not evenly spread: AMI alone contributes +2.83 pp of it. Excluding AMI, the far-field multi-party meeting set, the mean of the remaining seven moves from 6.39% to 6.67%, a +0.28 pp difference. For near-field single-speaker dictation, which is what this artifact is packaged for, the relevant number is the smaller one.

Rekody has not re-run the Open ASR Leaderboard at this profile, so no Rekody-measured leaderboard table is published here. The 560 ms repository's table should not be read as applying to this artifact.

Why trust this conversion

• Produced by the same pipeline as the 560 ms artifact, with --right-context as the only changed input: the same pinned checkpoint revision, the same NeMo 2.7.3 export script, and the same onnxruntime 1.23.2 quantize_dynamic QInt8 call over MatMul and LSTM ops.
• The export was gated on parity against NeMo's own streaming step on the same inputs: max absolute difference 0.0.
• decoder_joint.onnx came out bit-identical to the 560 ms repository's (89cae615… before metadata stamping). The RNN-T decoder does not depend on the encoder's latency profile, so it must not change — and it did not, which is a direct check that the pipeline reproduced the earlier conversion.
• tokenizer.model is byte-identical to the 560 ms repository's.
• The quantization coverage matches the 560 ms artifact exactly: 217 MatMulInteger, 170 DynamicQuantizeLinear, 0 ConvInteger, 77 float Conv in the encoder; 2 DynamicQuantizeLSTM and 3 MatMulInteger in the decoder, with the embedding table left fp32.
• The cost of int8 quantization on this architecture was measured as statistically zero on the 560 ms export: on LibriSpeech test-clean (n = 2620), int8 differs from the fp32 export it came from by +0.072 pp WER with a paired bootstrap 95% CI of [-0.004, +0.150], which includes zero.

License

Two licenses apply to this repository, and users must comply with both:

1. The conversion work (the int8 quantization, packaging, and benchmarking published here) is licensed under the PolyForm Shield License 1.0.0 (see LICENSE). PolyForm Shield is a noncompete license: any use is permitted except providing a product that competes with the software or with the licensor's products built on it.
2. The base model weights remain licensed by NVIDIA Corporation under the NVIDIA Open Model License (see NOTICE and the dated copy in licenses/).

File integrity (SHA-256)

These hashes are also published in SHA256SUMS.

Provenance

• Source: nvidia/nemotron-speech-streaming-en-0.6b, revision df1f0fe9dfdf05152936192b4c8c7653d53bf557 (the March 13, 2026 checkpoint).
• Conversion recipe: NeMo 2.7.3 ONNX export at att_context_size = [70, 1], then onnxruntime 1.23.2 quantize_dynamic QInt8 over MatMul and LSTM ops, then the streaming profile stamped into the encoder's ONNX metadata.

Contact: [email protected]