Downloads · 30 days
31
18% of all-time downloads
NemotronLabs-VoiceChat-11B-JANG_2
OsaurusAI/NemotronLabs-VoiceChat-11B-JANG_2
NemotronLabs-VoiceChat-11B-JANG_2 is a audio-text-to-text model from OsaurusAI. Use it for the audio-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as other.
<p align="center"<a href="https://osaurus.ai"<img src="./osaurus-x-banner.png" alt="Osaurus AI"</a</p
All-time downloads
175
Public
Parameters
11.1B
8 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors6.1 GB · 100%
Parameter types
How the weights are stored.
U3210.7B · 96%
Base models
Model card
From the Hugging Face model README
<p align="center"><a href="https://osaurus.ai"><img src="./osaurus-x-banner.png" alt="Osaurus AI"></a></p>
OsaurusAI/NemotronLabs-VoiceChat-11B-JANG_2
JANG_2 MLX bundle of nvidia/NVIDIA-NemotronLabs-VoiceChat-11B — smallest tier — 2-bit backbone, with the speech path deliberately protected.
VoiceChat is a full-duplex speech model: it listens on a continuous audio timeline, transcribes, answers in text, and synthesizes aligned speech — while handling interruptions, turn-taking, and tool calls.
Bundle
| Field | Value |
|---|---|
| Size on disk | 5.71 GiB |
| Source precision | fp32 (41 GiB) -> bf16 (21 GiB) -> this |
| Architecture | nemotron_voicechat |
| Backbone | nemotron_h hybrid — 56 layers (27 Mamba2 / 25 MLP / 4 attention) |
| Hidden size | 4480 |
| Audio in | 16000 Hz mono |
| Audio out | 22050 Hz, speaker Aria |
| Frame duration | 0.08s (12.5 fps) |
| Perception | Conformer 24L d=1024, causal streaming |
| Speech codec | 31x RVQ, codebook 1024 |
| Shards | 2 |
| Bit distribution | {2: 491, 3: 42, 4: 15, 6: 4, 8: 11} |
How it was quantized
Three calibration methods, all driven by one capture pass over real conversational audio (562 modules, 3.38 M row-samples, exercising speech-in AND speech-out):
| Method | Detail |
|---|---|
| Hessian-trace allocation | bits assigned by measured tr(H)·‖W‖²_F per module, not by tensor name |
| AWQ | alpha=0.15, salient-channel scaling absorbed into the producing RMSNorm across 112 norm groups / 204 projections |
| imatrix refit | activation-weighted affine fit replacing RTN — mean weighted rel-err 0.2145 |
Measured sensitivity drove the floors — the TTS subword encoder turned out to be the most sensitive component in the whole model (8x the mean trace of lm_head), and lm_head / function_head measured identically, so they get equal width.
Tensors deliberately kept at full precision
| Tensor | Why |
|---|---|
tts_model.rvq_embs | RVQ codebook — lookup entries, not a projection. Quantizing moves every centroid and corrupts all decoded audio. |
audio_prompt_latents.* | Speaker identity (83 KiB). Also what custom voices are made of. |
mog_head.proj_mus | read raw (reshaped, not called), so a packed weight breaks the mixture head |
embed_subword.embed_tokens | its dtype is read to allocate a buffer; quantized it would be uint32 and silently truncate floats |
Modalities
| Capability | Status |
|---|---|
| Speech in (ASR) | supported — RNN-T transducer, 16 kHz mono |
| Text out | supported |
| Speech out | supported — 22.05 kHz, built-in Aria voice |
| Tool calling | separate function_head channel — a full vocab-sized head, not text parsed after the fact |
| Full duplex | model supports interruptions / turn-taking; see Verification below for what has actually been exercised |
Verification
Verified end-to-end on this bundle, offline turn from real audio:
- loads and runs
- ASR transcript correct and identical to the bf16 reference
- coherent, on-topic reply text
- speech output objectively speech-shaped — checked by RMS, zero-crossing rate and spectral centroid, not merely "a wav was produced" (silence and noise both produce wavs)
- protected tensors verified byte-identical to the source
🚨 Not yet verified on this bundle: barge-in, turn-taking, and the tool channel under live duplex operation. Those require a streaming duplex session and have not been exercised here. Do not assume them.
Throughput is deliberately not quoted — it depends on host, runtime and settings, and a single number on a card is misleading.
Usage
Requires a runtime that implements nemotron_voicechat (upstream mlx-vlm >= 0.6.15).
from mlx_vlm.models.nemotron_voicechat import Model, ModelConfig, VoiceChatSession
# NOTE: a quantized bundle must be nn.quantize()d BEFORE load_weights,
# using the per-module map in config.json['quantization'] — otherwise
# packed uint32 codes are read as ordinary weights.
A worked loader, fixture prep and speech-comparison script are in the JANG examples.
Credits
JANG quantization by Jinho Jang — [email protected]
Base model by NVIDIA NemotronLabs. MLX layout derived from mlx-community/NemotronLabs-VoiceChat-11B-bf16.
- Task
- Audio-Text-to-Text
- Library
- mlx
- Type
- nemotron_voicechat
- License
- other
- Languages
- en
- Created
- Aug 21, 2026
- Updated
- Aug 31, 2026