Downloads · 30 days
34
0% of all-time downloads
FreedomAISVR/Nemotron-3-30B-Nano-Omni-NVFP4-GGUF
Nemotron-3-30B-Nano-Omni-NVFP4-GGUF is a text generation model from FreedomAISVR. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as other.
NVFP4 (E4M3) quantization of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, NVIDIA's 30B MoE (~3B active) reasoning model with Mamba2-SSM hybrid architecture.
Downloads · 30 days
34
0% of all-time downloads
All-time downloads
11.2K
Public
Repo size
19.6 GB
Likes
0
Public
Click a slice to open those files.
.gguf20.8 GB · 100%
From the Hugging Face model README
NVFP4 (E4M3) quantization of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, NVIDIA's 30B MoE (~3B active) reasoning model with Mamba2-SSM hybrid architecture.
This model uses a Mamba2-Transformer hybrid MoE architecture — 23 out of 52 layers are Mamba2 state-space model (SSM) layers. The CUDA kernels that run Mamba2 SSM operations (SSM_SCAN, SSM_CONV) require F32 inputs and will reject quantized weight tensors entirely.
When SSM weight tensors are quantized, the CUDA backend falls back to CPU for those layers. Since 44% of the model is Mamba2, this significantly impacts performance.
The 46 SSM weight tensors (ssm_in.weight, ssm_out.weight across 23 Mamba2 blocks) are stored in F16 instead of the quantized format. Everything else stays at the original quantization.
This hybrid approach is a temporary workaround until llama.cpp adds quantized support to CUDA SSM kernels (SSM_SCAN, SSM_CONV). The kernels are currently hardcoded to F32 (ssm_conv_f32, ssm_scan_f32 in the CUDA backend).
| Filename | Type | Size | Description |
|---|---|---|---|
nemotron-3-30b-NVFP4.gguf | Hybrid NVFP4+F16 | ~18.4 GB | Main model weights (~4.88 BPW effective) |
mmproj-nemotron-3-30b-f16.gguf | F16 | ~1.5 GB | SigLIP vision encoder |
llama-server -m nemotron-3-30b-NVFP4.gguf --mmproj mmproj-nemotron-3-30b-f16.gguf --host 0.0.0.0 --port 8080 -ngl 99