VoiceChat-11B-Perception-MLX-int5

VoiceChat 11B — MLX 5-bit (output heads 8-bit)

NVIDIA's complete duplex speech-to-speech NemotronLabs VoiceChat 11B pipeline, converted to MLX for Apple Silicon: continuous user-speech perception, the Nemotron-H text/function channels, direct EAR-TTS agent-speech generation, and the audio-codec decoder.

The upstream NVIDIA model card cites SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model (Interspeech 2025) as its primary architecture reference.

The repository name retains Perception for compatibility with the original understanding-only release. The current payload is the complete encoder/ + llm/ + tts/ bundle; strict full-pipeline loaders reject an older download that does not contain tts/.

Scope. Built-in Aria speech generation is verified end to end. The codec encoder used for voice cloning is included for checkpoint completeness but is not verified and does not currently reconstruct real audio correctly. Use the shipped Aria prompt; do not advertise custom-voice cloning.

Performance status. The runtime is functionally streaming, but the full speech-to-speech path does not yet sustain the 80 ms frame budget on the tested Apple Silicon system. Model-timeline turn positions are not wall-clock latency. Natural turn-taking latency has not yet been established against an upstream reference, so forced-BOS regression timings must not be cited as natural latency.

The current Swift release benchmark on an M5 Pro (48 GB) feeds one 80 ms live input frame per push over a controlled 120-frame fixture:

Both variants remain slower than real time once perception is included. The first-token/audio figures are hot compute latency for the first spoken response frame, not learned turn-taking latency. Peak macOS physical footprint was 24.42 GB for INT8 and 20.94 GB for INT5; it includes file-backed MLX mappings that RSS can undercount.

Model

Files

Performance

Measured against the fp16 bundle these were quantized from, teacher-forced over an identical corpus. Agreement is how often the quantized model picks the same next token as fp16; KL is the divergence of the full next-token distribution (lower is better for both).

This variant: 92.55% agreement, KL 0.01213.

8-bit reproduces the fp16 model's greedy output token for token. 5-bit diverges on roughly one token in thirteen — acceptable for conversational text, but worth measuring on your own task before relying on it for structured output such as tool-call arguments, where a single divergent token invalidates the result.

Perplexity is deliberately not quoted: on a short corpus, quantization noise can lower it without the model being better, so agreement and KL are the honest measures here.

Usage

Swift — complete duplex speech-to-speech session

import VoiceChat

let model = try await VoiceChatModel.loadFromHub(
    "aufklarer/VoiceChat-11B-Perception-MLX-int5")
let session = try await model.startSession()

for event in try await session.pushAudio(mono16kSamples) {
    playback.enqueue(
        event.audio,
        sampleRate: VoiceChatSession.outputSampleRate)
}

// Let a response that starts near the end of a finite clip finish.
_ = try await session.pushSilence(seconds: 6)
print(await session.userTranscript())
print(await session.reply())

Each event contains the text/function decisions and exactly one 1,764-sample, 22.05 kHz output frame. EAR-TTS uses the checkpoint's text control roles and eight MaskGIT unmasking iterations; it is not a projection from the 4,480-wide language-model hidden state.

Python / MLX

import mlx.core as mx
from voicechat_mlx import load_llm, load_perception

# Speech understanding: log-mel -> language-model embedding space
encode = load_perception("encoder")
embeddings, lengths = encode(log_mel)     # (B, T, 128) -> (B, T/8, 4480)

# Language backbone
model, tokenizer = load_llm("llm")
ids = mx.array(tokenizer.encode("The capital of Norway is"))
logits = model(ids[None])
print(tokenizer.decode([int(mx.argmax(logits[0, -1]))]))

pip install mlx mlx-lm parakeet-mlx huggingface_hub
hf download aufklarer/VoiceChat-11B-Perception-MLX-int5 --local-dir ./voicechat
python -c "from voicechat_mlx import load_llm; m, t = load_llm('./voicechat/llm'); print('ok')"

Generate the built-in Aria voice, or render both sides of a duplex exchange:

python speak.py --checkpoint . --tokenizer llm --text "Hello from VoiceChat" --out hello.wav
python render_dialog.py --bundle . --audio user.wav --out dialog.wav

Every published speech payload must first pass the canonical silence check:

python codec_mlx.py --checkpoint . --verify-silence

The encoder is a NeMo streaming FastConformer and differs from a stock Conformer in three ways that all fail silently if ignored — no biases on the feed-forward, attention and convolution linears; a LayerNorm in place of the convolution BatchNorm; and causal subsampling that keeps 17 frequency bins rather than 16. voicechat_mlx.py handles all three, so use it rather than constructing the module tree yourself.

Source

Converted from nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The language backbone derives from nvidia/NVIDIA-Nemotron-Nano-9B-v2, which also supplies the tokenizer. Licensed under OpenMDW 1.1.

Links

• speech-swift — Apple SDK
• Docs — install and CLI docs
• soniqo.audio — website
• blog — blog
• Speech Studio — local speech generation app