NemotronLabs-VoiceChat-11B-mlx-4bit

NemotronLabs-VoiceChat-11B — MLX (4bit)

MLX conversion of nvidia/NVIDIA-NemotronLabs-VoiceChat-11B, a duplex speech-to-speech model that listens and speaks at the same time. 11.1 B parameters, converted from the original float32 release (44.4 GB) for Apple Silicon.

What is inside

The conversion keeps all four components of the model, complete and in the original layout.

Duplex operation runs at 80 ms frames (12.5 frames/s), 16 kHz input and 22.05 kHz output.

What runs in MLX today

Two parts of the model are directly usable, with runnable examples below:

• the language backbone, via mlx-lm's nemotron_h implementation
• the audio codec — waveform to residual-VQ codes and back

The Conformer speech encoder and the full duplex loop are not yet implemented in MLX. The weights for them are present and complete in this repository.

Example 1 — text generation with the language backbone

pip install mlx mlx-lm
python mlx/extract_llm.py --src . --dst ./voicechat-llm
python mlx/chat_example.py --model ./voicechat-llm --prompt "The capital of France is"

The source model ships no text tokenizer, so extract_llm.py pairs the backbone with a Nemotron-H base tokenizer (vocabulary 131072) and verifies the size matches before writing.

Example 2 — audio codec round trip

pip install mlx numpy
python mlx/codec_example.py --repo . --wav input.wav --out output.wav

Encodes a waveform to 512-dimensional latents and 31-stage codes, then decodes back to audio. Reconstruction is about 8 dB SNR at 12.5 frames per second, and runs roughly 6x faster than real time on an M4 Max.

Quantization

The language backbone and the three vocabulary heads are quantized to 4 bits (group size 64) — 9.47 B of the 11.10 B parameters. The entire speech path is left in bfloat16: the audio codec, the mixture-of-Gaussians head, the Conformer encoder and the RNN-T decoder. Those are small tensors whose error would land directly on audio quality for a negligible size saving.

Mean relative error against the bf16 tier, measured per tensor: 9.69 %.

Measured performance

Apple M4 Max, 68.7 GB unified memory, language backbone only, greedy decoding:

Audio codec: about 6x real time.

Notes

• Text-only prompting is outside the model's training distribution. It was trained to emit interleaved text and audio-codec tokens, so generations may end with audio-side tokens such as <SPECIAL_12>.
• The codec expects 22.05 kHz mono audio. Other rates decode at the wrong speed.
• Every tensor from the source checkpoint is present; the conversion is verified for completeness and for finite values in all tiers.

License and attribution

Released under OpenMDW-1.1, following the original model. Base model and architecture by NVIDIA; see nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The MLX audio-codec implementation follows NVIDIA NeMo Speech.