parakeet-tdt-0.6b-v3-onnx

Parakeet TDT 0.6B v3 - ONNX (Quantized)

ONNX exports of NVIDIA Parakeet TDT 0.6B ASR encoder in multiple precision formats.

Model Variants

INT8 cosine measured on random noise input; real audio quality expected to be higher. FP8 and INT4 models use custom weight formats requiring DequantizeLinear or custom loader.

Files

• config.yaml - Model configuration
• decoder_joint.onnx - Decoder/Joint network (shared across all variants)
• tokenizer/ - SentencePiece tokenizer files
• fp32/, fp16/, int8/, fp8/, int4/ - Encoder variants

Usage

import onnxruntime as ort
import numpy as np

# FP16 (recommended for GPU) or INT8 (recommended for CPU)
sess = ort.InferenceSession("fp16/encoder.onnx", providers=["CUDAExecutionProvider"])

audio_features = np.random.randn(1, 128, 100).astype(np.float32)
length = np.array([100], dtype=np.int64)

outputs = sess.run(None, {"audio_signal": audio_features, "length": length})
encoded = outputs[0]  # [batch, d_model, time]

Quantization Details

• FP16: Full graph + weight conversion via onnxruntime.transformers.optimizer
• INT8: Dynamic weight-only quantization via onnxruntime.quantization.quantize_dynamic
• FP8: Manual weight conversion to E4M3FN format using ml_dtypes
• INT4: Symmetric per-tensor INT4 quantization, packed 2 values per byte, with JSON scale metadata

Source

Exported from NVIDIA NeMo Parakeet TDT 0.6B checkpoint on DGX Spark (GB10 Grace Blackwell).