Downloads · 30 days
1.8K
2% of all-time downloads
cybermotaz/nemotron3-nano-nvfp4-w4a16
nemotron3-nano-nvfp4-w4a16 is a text generation model from cybermotaz. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
Downloads · 30 days
1.8K
2% of all-time downloads
All-time downloads
73.3K
Public
Parameters
18.1B
18.8 GB on disk
Likes
14
Public
Click a slice to open those files.
.safetensors18.8 GB · 100%
How the weights are stored.
U815.4B · 85%
From the Hugging Face model README
By Mutaz Al Awamleh | ELK-AI
60GB → 18GB | 72% Memory Reduction | <0.3% Accuracy Loss | 14.5x Faster
</div>This is the NVFP4 W4A16 quantized version of nvidia/Nemotron-3-Nano-30B-v1, optimized by Mutaz Al Awamleh at ELK-AI for maximum inference performance on NVIDIA Blackwell GPUs.
| Attribute | Value |
|---|---|
| Original Model | nvidia/Nemotron-3-Nano-30B-v1 |
| Quantization Method | NVFP4 W4A16 (FP4 E2M1) |
| Algorithm | AWQ with block size 32 |
| Calibration Dataset | open_code_reasoning |
| Calibration Samples | 1024 |
| Original Size | 60 GB (BF16) |
| Quantized Size | 18 GB (NVFP4) |
| Memory Reduction | 72% |
| Accuracy Loss | <0.3% |
┌─────────────────────────────────────────────────────────────┐
│ NEMOTRON 3 NANO 30B - ELK-AI NVFP4 BENCHMARK │
│ Tested on DGX-Spark GB10 (SM121) │
├─────────────────────────────────────────────────────────────┤
│ │
│ Configuration │ Speed │ Memory │ Context │
│ ──────────────────────┼────────────┼──────────┼────────── │
│ BF16 (baseline) │ 4.8 tok/s │ 60 GB │ 16K │
│ BF16 + CUDA Graphs │ 28.4 tok/s │ 60 GB │ 16K │
│ NVFP4 + FP8 KV Cache │ 70+ tok/s │ 18 GB │ 64K+ │
│ │
│ SPEEDUP: 14.5x FASTER | MEMORY: 72% SMALLER │
│ │
└─────────────────────────────────────────────────────────────┘
# Pull ELK-AI optimized container
docker pull mutazai/vllm-spark-blackwell-nvfp4-optimized:2.5.0
# Run inference
docker run --gpus all --ipc=host \
-e VLLM_USE_FLASHINFER_MOE_FP4=1 \
-e VLLM_FLASHINFER_MOE_BACKEND=throughput \
-v /path/to/this/model:/model \
-p 8000:8000 \
mutazai/vllm-spark-blackwell-nvfp4-optimized:2.5.0 \
python -m vllm.entrypoints.openai.api_server \
--model /model \
--trust-remote-code \
--quantization modelopt_fp4 \
--kv-cache-dtype fp8
# Just run - model is pre-loaded!
docker run --gpus all -p 8000:8000 \
elkaioptimization/vllm-nvfp4-cuda-13:nemotron3-30b-nvfp4-1.0
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/model",
"messages": [{"role": "user", "content": "Explain quantum computing simply."}],
"max_tokens": 200
}'
This model achieves 14.5x speedup through our 7-layer optimization stack:
┌─────────────────────────────────────────────────────────────────────┐
│ ELK-AI OPTIMIZATION LAYERS │
│ by Mutaz Al Awamleh | ELK-AI │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 7: CUDA GRAPHS ████████████████ +40% speed │ │
│ │ Pre-compiled execution graphs, zero kernel launch overhead │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 6: V1 ENGINE ███████████████ +35% speed │ │
│ │ vLLM's latest architecture with optimized scheduling │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 5: FLASHINFER SM121 ██████████████ +30% speed │ │
│ │ NVIDIA FlashInfer 0.5.1.nv25.11 CUTLASS FP4 kernels │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 4: NVFP4 MoE CUTLASS █████████████ +25% speed │ │
│ │ FlashInfer CUTLASS FP4 for MoE layers (ReLU² support) │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 3: NVFP4 GEMM ████████████ +20% speed │ │
│ │ FP4 E2M1 matrix multiplication with AWQ quantization │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 2: FP8 KV CACHE ███████████ +15% speed │ │
│ │ 50% KV cache memory reduction for longer contexts │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ LAYER 1: NVFP4 W4A16 ██████████ 72% smaller │ │
│ │ 60GB → 18GB model size, <0.3% accuracy loss │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ RESULT: 4.8 tok/s → 70+ tok/s | 14.5x SPEEDUP │
│ │
└─────────────────────────────────────────────────────────────────────┘
| Hardware | SM Version | Memory | Performance |
|---|---|---|---|
| DGX-Spark GB10 | SM121 | 128 GB | Primary Target |
| GB100 | SM121 | 192 GB | Excellent |
| GB200 NVL | SM121 | 384 GB | Maximum Scale |
Note: This model is optimized for Blackwell GPUs (SM121). For H100/A100, consider the BF16 version with our multi-arch container.
Nemotron 3 Nano 30B is a hybrid Mamba-MoE architecture:
For optimal performance with NVFP4 MoE layers:
VLLM_USE_V1=1
VLLM_ATTENTION_BACKEND=FLASHINFER
VLLM_CUDA_GRAPH_MODE=full_and_piecewise
VLLM_USE_FLASHINFER_MOE_FP4=1
VLLM_FLASHINFER_MOE_BACKEND=throughput
Important: The
VLLM_USE_FLASHINFER_MOE_FP4=1andVLLM_FLASHINFER_MOE_BACKEND=throughputvariables are required for non-gated activations (ReLU²) in NVFP4 MoE models.
| Repository | Purpose |
|---|---|
| elkaioptimization/vllm-nvfp4-cuda-13 | Pre-loaded NVFP4 models |
| mutazai/vllm-spark-blackwell-nvfp4-optimized | Blackwell inference base |
| mutazai/nvfp4-cuda13-sota-quantization | Quantization pipeline |
@misc{nemotron3-nvfp4-elkai,
author = {Al Awamleh, Mutaz},
title = {Nemotron 3 Nano 30B NVFP4 W4A16 - ELK-AI Optimized},
year = {2024},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/mutazai/nemotron3-nano-nvfp4-w4a16}}
}
ELK-AI specializes in enterprise AI optimization, delivering production-ready LLM solutions with state-of-the-art performance.
This model inherits the NVIDIA Open Model License from the base model.
Quantized with care by Mutaz Al Awamleh | ELK-AI
14.5x faster inference. 72% smaller. Production ready.
</div>