parakeet-realtime-eou-120m-streaming-fp16

Parakeet Realtime EOU 120M v1 — cache-aware ONNX (FP16)

This is an ONNX export of NVIDIA's parakeet_realtime_eou_120m-v1 FastConformer RNN-T model with the cache-aware encoder I/O ports preserved, so it can be used for streaming inference under ONNX Runtime without mid-word corruption at chunk boundaries.

It is intended as a drop-in replacement for the community export at ysdede/parakeet-realtime-eou-120m-v1-onnx for users who need the cache-aware streaming surface.

Why this export exists

The encoder is a 17-layer FastConformer with attention context [70, 1] (5.6 s left, 80 ms right). When that left-context state is not threaded across chunks, the encoder warms up mid-word at every chunk boundary, producing artifacts like talk / ovilla / c for the word "talkative". The community ONNX export strips the cache I/O ports, which is fine for full-utterance offline use but breaks streaming.

This export wraps cache_aware_stream_step() so all four cache tensors are exposed as explicit ONNX inputs and outputs, allowing the caller to thread state across chunks. The wrapper pattern is taken from FluidAudio's convert_streaming_encoder.py, translated from CoreML to ONNX.

Files

Cache-aware I/O contract

The streaming encoder takes six inputs and produces six outputs (cache shapes are for batch=1):

On the first chunk of an utterance, all four cache tensors are zero-initialized. On every subsequent chunk, the caller passes the previous step's new_* outputs back in as the corresponding inputs.

The decoder + joint network is stateless per-call at the encoder level (the LSTM state is threaded via its own existing input_states_1 / input_states_2 inputs), so the ysdede decoder ONNX is reused unchanged.

Numerical fidelity

Compared to NVIDIA's PyTorch reference (cache_aware_stream_step()), running 11 sequential cache-threaded chunks of real audio:

• FP32 export → PyTorch: max absolute difference 3.6 × 10⁻⁵ (~30× tighter than the 1 × 10⁻³ tolerance from the parity check).
• FP16 export → FP32 export: max relative error ~0.5% on intermediate activations; downstream RNN-T greedy decoding produces character-identical output for the validation clips.

End-to-end validation

Running the cache-aware encoder against two real desktop-recorded clips and comparing to the same model loaded with the cache I/O stripped:

Note: extensions survives as one whole word in the cache-aware output, where the stateless export splits it as these / ven / ans at chunk boundaries.

Usage

A minimal cache-threading loop in Python:

import numpy as np
import onnxruntime as ort

sess = ort.InferenceSession("streaming_encoder.fp16.onnx", providers=["CPUExecutionProvider"])

# Initial caches: all zeros.
pre_cache = np.zeros((1, 128, 16), dtype=np.float32)
cache_lc = np.zeros((17, 1, 70, 512), dtype=np.float32)
cache_lt = np.zeros((17, 1, 512, 8), dtype=np.float32)
cache_len = np.zeros((1,), dtype=np.int32)

for mel_chunk in stream_of_mel_features:           # each is [1, 128, 128]
    feeds = {
        "audio_signal": mel_chunk,
        "audio_length": np.asarray([128], dtype=np.int32),
        "pre_cache": pre_cache,
        "cache_last_channel": cache_lc,
        "cache_last_time": cache_lt,
        "cache_last_channel_len": cache_len,
    }
    encoded, enc_len, pre_cache, cache_lc, cache_lt, cache_len = sess.run(None, feeds)
    # ...feed `encoded` (shape [1, 512, 16]) into the decoder+joint network...

License and attribution

This export is derived from NVIDIA's parakeet_realtime_eou_120m-v1 checkpoint and is therefore released under the NVIDIA Open Model License (June 2024). The wrapper pattern is adapted from FluidAudio's convert_streaming_encoder.py, also under the NVIDIA Open Model License.

• Original model: https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1
• Stateless community ONNX (decoder reused from this): https://huggingface.co/ysdede/parakeet-realtime-eou-120m-v1-onnx
• CoreML cache-aware export and wrapper code: https://huggingface.co/FluidInference/parakeet-realtime-eou-120m-coreml
• License: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf