nemotron-3.5-asr-streaming-0.6b-int8-mlx

Nemotron 3.5 ASR Streaming 0.6B — MLX (int8)

Pure-MLX int8 conversion of NVIDIA's Nemotron 3.5 ASR Streaming 0.6B for cache-aware multilingual transcription on Apple Silicon. Linear and embedding layers use 8-bit affine quantization with group size 64. The complete waveform-to-text runtime uses MLX; PyTorch, NeMo, and Transformers are not imported during inference.

This is the only published Nemotron build and the default in mlx-speech. A temporary pre-release comparison over 44 minutes of ordinary English and Mandarin web audio found negligible accuracy differences from the local bf16 validation reference: +0.035 percentage points English WER and +0.112 points Mandarin CER.

Model details

• Upstream developer: NVIDIA
• MLX conversion and runtime: App Automaton
• Architecture: 24-layer cache-aware FastConformer encoder, language prompt, greedy RNN-T decoder
• Parameters: approximately 0.6B
• Precision: int8 affine Linear/Embedding weights, group size 64. Convolutions, normalization, biases, recurrent parameters, unsupported shapes, and other non-eligible tensors retain their native precision; preprocessing buffers remain fp32.
• Input: mono audio, resampled to 16 kHz by the public file adapter
• Runtime: batch one on Apple Silicon

Usage

pip install mlx-speech

import mlx_speech

asr = mlx_speech.asr.load("nemotron-asr-streaming")
result = asr.generate(
    "speech.wav",
    language="en-US",
    att_context_size=(56, 13),
)
print(result.language, result.text)

For a live waveform stream:

session = asr.stream_session(language="en-US", att_context_size=(56, 3))
for pcm_chunk in microphone:
    emitted_token_ids = session.feed(pcm_chunk)
    consume(emitted_token_ids)
session.finalize()
print(session.result().text)

feed() accepts arbitrary sample counts. finalize() is required to flush centered-STFT, subsampling, encoder, and RNN-T state.

Latency modes

One encoder frame is 80 ms. The left context is 56 frames; right context selects the trained latency/accuracy point without retraining. Tier-1 WER below is reported by NVIDIA for the upstream model.

(56, 13) is the default accuracy setting. PCM packet size is independent of model context and does not change the emitted tokens.

Language tiers

The checkpoint recognizes 40 locales, but they are not one quality tier.

The adaptation-ready locales are tokenizer-recognized fine-tuning targets, not production-ready zero-shot transcription claims. Use an explicit locale when known; language="auto" is supported but can reduce accuracy.

Pre-release accuracy comparison

Before choosing the release artifact, bf16 and int8 were run at (56, 13) on the same four longer recordings with human-provided transcripts. The inputs and detailed hypotheses were kept in a temporary system-cache directory and are not shipped as a benchmark or committed to mlx-speech.

The 755,732,373-byte int8 weight file is 40.8% smaller than the local 1,276,192,217-byte bf16 validation artifact. Arbitrarily ragged waveform feeds produce the same tokens as offline int8 inference, and cache storage remains bounded with utterance length.

Limitations

• batch size one
• greedy RNN-T only; no beam search
• no word timestamps or forced alignment
• inference requires Apple Silicon and MLX
• the pre-release comparison covers English and Mandarin only and is not a substitute for NVIDIA's broader upstream evaluation

Links

• Source code: appautomaton/mlx-speech
• Project page: appautomaton.com/mlx-speech

License and attribution

The model weights are governed by OpenMDW-1.1. The complete license text is included in this repository as LICENSE.OpenMDW-1.1. This is a quantized derivative of nvidia/nemotron-3.5-asr-streaming-0.6b; NVIDIA developed and released the upstream model. The mlx-speech runtime code is separately licensed under MIT.