nemotron-speech-streaming-en-0.6b-int8

Nemotron Speech Streaming EN 0.6B — ONNX int8 (560 ms)

Dynamic int8-quantized ONNX export of nvidia/nemotron-speech-streaming-en-0.6b for use with parakeet-rs.

English-only, cache-aware streaming ASR (~600M parameters). Preserves disfluencies (um, uh) and outputs verbatim English with punctuation.

This bundle uses the 560 ms streaming mode (--right-context 6 at export, the default NVIDIA preset). Other latencies (80 / 160 / 1120 ms) require a separate export and matching CHUNK_SIZE in parakeet-rs.

Files

All files must live in the same directory:

No config.json is required at inference time — parakeet-rs embeds streaming constants in Rust.

Size vs fp32

Roughly 75% smaller than the fp32 ONNX export.

Model details

How this was produced

1. Export fp32 ONNX at 560 ms latency:

python parakeet-rs/scripts/export_nemotron_streaming.py \
  nemotron-speech-streaming-en-0.6b/nemotron-speech-streaming-en-0.6b.nemo \
  out_onnx/ \
  --right-context 6

1. Quantize to int8:

pip install onnx 'onnxruntime>=1.20'

python parakeet-rs/scripts/quantize_nemotron_streaming.py \
  out_onnx/ \
  out_onnx_int8/ \
  --clean-output

Quantization uses ONNX Runtime dynamic int8 (quantize_dynamic): encoder QUInt8, decoder/joint QInt8.

Streaming latency presets (export only)

Latency is not selectable in the quantize script or Nemotron::from_pretrained — re-export fp32, re-quantize, and update CHUNK_SIZE in parakeet-rs/src/nemotron.rs.

Usage (parakeet-rs)

Nemotron::from_pretrained takes the model directory and an optional execution provider only. Chunking is done via transcribe_chunk, which buffers audio until 560 ms of new audio is ready (for this export).

use parakeet_rs::Nemotron;

let mut model = Nemotron::from_pretrained("./out_onnx_int8", None)?;

// Feed any slice size; encoder runs every 560 ms of new audio.
for frame in audio_source.frames() {
    let text = model.transcribe_chunk(&frame)?;
    print!("{}", text);
}

// Or feed 560 ms blocks directly (8960 samples @ 16 kHz)
const FEED_560MS: usize = 8960;
for chunk in audio.chunks(FEED_560MS) {
    let text = model.transcribe_chunk(chunk)?;
    print!("{}", text);
}

Example:

cd parakeet-rs
cargo run --release --example streaming /path/to/audio.wav
# Point the example at this directory (./nemotron or pass path)

The loader picks encoder.int8.onnx and decoder_joint.int8.onnx when present. This repo build sets CHUNK_SIZE = 56 for the 560 ms export.

Requirements

• Inference: parakeet-rs with ONNX Runtime
• Re-export / re-quantize: Python 3.10+, NeMo (export only), onnx, onnxruntime>=1.20

Quality notes

• Dynamic int8 quantizes weights only; activations stay float32 at runtime.
• Run WER checks on your audio before production deployment.
• ORT warnings about unquantizable Slice/Tile tensors during quantize are expected.

License

Derived from NVIDIA Nemotron Speech Streaming. Use and redistribution follow the license terms of the original model.