Downloads · 30 days
195
3% of all-time downloads
LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF
Nemotron-Labs-Audex-30B-A3B-GGUF is a text generation model from LordNeel. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as other.
GGUF quantizations of NVIDIA Nemotron-Labs-Audex-30B-A3B (source revision 48616f3d9d5ad960a4d03dc66f2178daf77e7b5f) — a 30B-parameter hybrid Nemotron-H MoE model (~3B active parameters per token, nemotronhmoe architec…
Downloads · 30 days
195
3% of all-time downloads
All-time downloads
6.6K
Public
Repo size
365 GB
Likes
9
Public
Click a slice to open those files.
.gguf391 GB · 98%
From the Hugging Face model README
GGUF quantizations of NVIDIA Nemotron-Labs-Audex-30B-A3B (source revision 48616f3d9d5ad960a4d03dc66f2178daf77e7b5f) — a 30B-parameter hybrid Nemotron-H MoE model (~3B active parameters per token, nemotron_h_moe architecture, 1,048,576-token context) with audio understanding and generation capabilities.
Every quant was measured for quality (BF16-referenced KL divergence / perplexity over three corpora) and throughput (llama-bench on 2× RTX PRO 6000 Blackwell Max-Q). Raw data lives in metrics/.
⚠️ License: NVIDIA OneWay Noncommercial. These files are derived from NVIDIA Nemotron-Labs-Audex-30B-A3B and are not for commercial use. Audex also includes or derives from third-party components with additional notices. See
LICENSE,license/NVIDIA-OneWay-Noncommercial-License.docx, andlicense/THIRD_PARTY_NOTICES.md.
| Track | Directory | What it is | Use with |
|---|---|---|---|
| Text-only | quants/ | GGUFs from the materialized Audex text-only checkpoint | Plain llama.cpp text generation |
| Audiogen | audio_quants/ | Full-vocab GGUFs for audio-token generation | The audio_support/ sidecar bundle |
The audio encoder/vocoder assets are not embedded as GGUF tensors — audio workflows need the audio_support/ sidecar (NV-Whisper + audio preprocessor path, causal speech decoder, enhancement VAE, and NVIDIA's HF/vLLM inference scripts).
For plain text work, use quants/. For audio-token generation, use audio_quants/ + audio_support/.
| Use case | File | Why |
|---|---|---|
| Near-lossless | Q8_0 | Mean KLD 0.006 (text) / 0.008 (audio), ~97% top-token agreement |
| Recommended default | Q5_K_M / Q5_K_L | ~95.5% top-token agreement at ~24 GiB, faster than Q8_0 |
| Smallest solid quant | Q4_K_M / Q4_K_L | ~94.5–94.7% top-token agreement at ~23 GiB, fastest K-quants |
| Experimental | MXFP4_MOE (text-only) | Smallest file (16.7 GiB) and by far the fastest prompt processing, but measurably lower fidelity |
# Text-only quant
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
quants/audex-30b-a3b-textonly-Q5_K_M.gguf --local-dir ./audex-gguf
# Audiogen quant + audio sidecar bundle
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
audio_quants/audex-30b-a3b-audiogen-Q5_K_M.gguf --local-dir ./audex-gguf
hf download LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
--include "audio_support/*" --local-dir ./audex-gguf
Verify downloads against SHA256SUMS / AUDIO_SHA256SUMS (per-file .sha256 files are also next to each GGUF).
llama.cpp supports the nemotron_h language-model architecture used by these checkpoints. The chat template is embedded in the GGUF.
llama-server \
--hf-repo LordNeel/Nemotron-Labs-Audex-30B-A3B-GGUF \
--hf-file quants/audex-30b-a3b-textonly-Q5_K_M.gguf \
-ngl -1 -sm layer \
-c 16384
For audio generation, audio QA, and unified speech-to-speech flows, use NVIDIA's inference scripts shipped in the sidecar:
audio_support/inference_scripts_hf/ — HF transformers path (inference_hf.py, inference_example.sh)audio_support/inference_scripts_vllm/audiogen_scripts/ — vLLM TTS / text-to-audioaudio_support/inference_scripts_vllm/audioqa_scripts/ — vLLM audio QA server + clientaudio_support/inference_scripts_vllm/unified_s2s_scripts/ — cascaded speech-to-speech web demoNote: llama.cpp does not currently package the full HF NemotronHAudexForConditionalGeneration audio pipeline as a single all-in-one GGUF runtime.
quants/)Quality = mean over three eval slices (WikiText-2, code, GSM8K); throughput from llama-bench (settings below). BF16 baseline included for reference.
| Quant | Size GiB | BPW | Mean KLD | Same top-p | Prompt tok/s | Gen tok/s |
|---|---|---|---|---|---|---|
| Q8_0 | 31.3 | 8.51 | 0.00561 | 97.03% | 8,500 | 287.4 |
| Q6_K | 31.2 | 8.49* | 0.00660 | 96.85% | 8,518 | 290.1 |
| Q5_K_L | 24.3 | 6.62 | 0.01243 | 95.55% | 8,245 | 334.2 |
| Q5_K_M | 24.2 | 6.60 | 0.01268 | 95.50% | 8,194 | 333.9 |
| Q4_K_L | 22.9 | 6.24 | 0.01749 | 94.73% | 8,640 | 345.5 |
| Q4_K_M | 22.8 | 6.21 | 0.01799 | 94.73% | 8,579 | 345.2 |
| MXFP4_MOE | 16.7 | 4.55 | 0.04048 | 92.30% | 11,486 | 329.2 |
| (BF16 reference) | ~59 | 16 | 0 | 100% | 4,378 | 176.9 |
Q6_K_L is also included but is byte-identical to Q6_K (K-quant fallbacks make the L variant resolve to the same tensors).MXFP4_MOE is experimental (0 fallback tensors, so it hits its native 4.55 BPW).| Quant | WikiText-2 KLD | WikiText-2 same-top-p | Code KLD | Code same-top-p | GSM8K KLD | GSM8K same-top-p |
|---|---|---|---|---|---|---|
| Q8_0 | 0.00644 | 96.03% | 0.00609 | 97.27% | 0.00432 | 97.81% |
| Q6_K | 0.00762 | 95.74% | 0.00701 | 97.17% | 0.00517 | 97.64% |
| Q5_K_L | 0.01465 | 94.11% | 0.01244 | 95.87% | 0.01019 | 96.68% |
| Q5_K_M | 0.01524 | 94.03% | 0.01212 | 95.78% | 0.01069 | 96.70% |
| Q4_K_L | 0.02142 | 92.90% | 0.01704 | 95.23% | 0.01402 | 96.06% |
| Q4_K_M | 0.02165 | 92.95% | 0.01774 | 95.15% | 0.01459 | 96.10% |
| MXFP4_MOE | 0.04963 | 89.50% | 0.03931 | 93.17% | 0.03250 | 94.22% |
Base (BF16) perplexities: WikiText-2 10.97, code 2.96, GSM8K 3.38. Full data: metrics/kld_ppl_summary.tsv, metrics/kld_ppl_metrics.json.
audio_quants/)Full-vocab audiogen language-model path — use together with audio_support/.
| Quant | Size GiB | BPW | Mean KLD | Same top-p | Prompt tok/s | Gen tok/s |
|---|---|---|---|---|---|---|
| Q8_0 | 31.7 | 8.51 | 0.00754 | 96.92% | 8,339 | 276.5 |
| Q6_K | 31.6 | 8.49 | 0.00934 | 96.68% | 8,350 | 279.4 |
| Q5_K_L | 24.7 | 6.65 | 0.01599 | 95.33% | 8,044 | 319.4 |
| Q5_K_M | 24.6 | 6.60 | 0.01672 | 95.27% | 8,077 | 319.5 |
| Q4_K_L | 23.3 | 6.27 | 0.02307 | 94.52% | 8,526 | 329.3 |
| Q4_K_M | 23.1 | 6.22 | 0.02342 | 94.54% | 8,447 | 329.6 |
| (BF16 reference) | ~60 | 16 | 0 | 100% | 4,285 | 169.6 |
| Quant | WikiText-2 KLD | WikiText-2 same-top-p | Code KLD | Code same-top-p | GSM8K KLD | GSM8K same-top-p |
|---|---|---|---|---|---|---|
| Q8_0 | 0.01039 | 95.83% | 0.00758 | 97.18% | 0.00465 | 97.75% |
| Q6_K | 0.01302 | 95.42% | 0.00928 | 97.03% | 0.00572 | 97.58% |
| Q5_K_L | 0.02190 | 93.66% | 0.01528 | 95.74% | 0.01079 | 96.60% |
| Q5_K_M | 0.02339 | 93.56% | 0.01528 | 95.64% | 0.01150 | 96.62% |
| Q4_K_L | 0.03279 | 92.49% | 0.02128 | 95.09% | 0.01514 | 95.97% |
| Q4_K_M | 0.03266 | 92.59% | 0.02197 | 94.99% | 0.01564 | 96.04% |
Full data: metrics/audio_kld_ppl_summary.tsv, metrics/audio_kld_ppl_metrics.json.
audio_support/)Common sidecar for the audiogen GGUFs — 58 files, ~7.67 GB. Manifest: audio_support/audio_support_manifest.json; checksums: audio_support/SHA256SUMS.
| Component | Purpose |
|---|---|
checkpoint_folder_full/ | Audio tower weights (model-audio.safetensors), audio preprocessor, and the full Audex custom configuration/modeling code |
nv-whisper/ | NV-Whisper encoder used on the audio input path |
audex_causal_speech_decoder/ | Streaming causal speech decoder |
enhancement_VAE/ | XCodec RVQ4 enhancement VAE (48 kHz audio enhancement) |
inference_scripts_hf/, inference_scripts_vllm/ | NVIDIA reference scripts for audio generation, audio QA, and unified speech-to-speech |
Quality — BF16-referenced quant-fidelity via llama-perplexity --kl-divergence, 64 chunks × 512-token context per slice, over three corpora: WikiText-2 raw test, a Magicoder-style code slice, and a GSM8K text slice. Imatrix statistics: metrics/imatrix_statistics.txt; imatrix file: gguf/audex-30b-a3b-textonly.imatrix.gguf.
Throughput — llama-bench on 2× NVIDIA RTX PRO 6000 Blackwell Max-Q: prompt 512, generation 256, batch/ubatch 512, 3 repetitions, -ngl -1 -sm layer.
Quant plans — configs/quant_plan.json, configs/audio_quant_plan.json. Release manifests: release_manifest.json, audio_release_manifest.json.
Q6_K_L is byte-identical to Q6_K. Details are in the artifact summary TSVs and per-quant logs.nemotron_h LM; the full audio pipeline requires the sidecar + NVIDIA's HF/vLLM scripts.