parakeet-tdt-0.6b-v3-int4

Parakeet TDT 0.6B v3 — weight-only int4 ONNX for sherpa-onnx

A 4-bit weight-only quantization of NVIDIA's Parakeet TDT 0.6B v3 for sherpa-onnx, built from the fp32 export csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3. It is the model shipped by OpenRamble for Windows, a local dictation app.

Why not the stock int8 export

The published int8 export is dynamically quantized (DynamicQuantizeLinear + MatMulInteger / ConvInteger): activations are cut to 8 bits at run time, including in the convolutional front end that sees the spectrogram first. On a quiet or muffled microphone — a laptop microphone array peaking around −25 dBFS — it answered Russian speech with nothing or with English filler, while the fp32 export read the same recording correctly.

This build quantizes only the encoder's matrix weights (MatMulNBits, 4-bit, symmetric, blocks of 64, accuracy_level=4); activations stay fp32. Decoder and joiner are the unmodified fp32 files.

Measured on FLEURS ru dev (40 utterances, 710 words), sherpa-onnx 1.13.5, CPU:

Same speed as the int8 export; peak memory about the same.

Files

Requires an ONNX Runtime with the MatMulNBits contrib op (sherpa-onnx ≥ 1.13 / ORT ≥ 1.17).

Usage (sherpa-onnx, Python)

import sherpa_onnx
rec = sherpa_onnx.OfflineRecognizer.from_transducer(
    encoder="encoder.int4.onnx", decoder="decoder.onnx", joiner="joiner.onnx",
    tokens="tokens.txt", model_type="nemo_transducer", num_threads=4)

Reproduce

scripts/quantize-parakeet-int4.py in the OpenRamble for Windows repository downloads the fp32 export and produces exactly these files (deterministic; SHA-256 of the encoder is stable across runs).

License

CC-BY-4.0, as the original model. Credit: NVIDIA (model), k2-fsa / csukuangfj (ONNX export).