Downloads · 30 days
2.2K
30% of all-time downloads
NANI-Nithin/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF is a text generation model from NANI-Nithin. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, converted with llama.cpp for fast local inference on CPU/GPU.
Downloads · 30 days
2.2K
30% of all-time downloads
All-time downloads
7.3K
Public
Repo size
524 GB
Likes
1
Public
Click a slice to open those files.
.gguf524 GB · 100%
From the Hugging Face model README
GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, converted with llama.cpp for fast local inference on CPU/GPU.
| Quant | Use case |
|---|---|
| F16 | Full precision, reference quality |
| Q8_0 | Near-lossless, largest quant size |
| Q6_K | Very high quality, minimal loss |
| Q5_K_M / Q5_K_S | High quality, good balance |
| Q4_K_M / Q4_K_S | Recommended default — best speed/quality tradeoff |
| Q4_0 | Legacy 4-bit, faster on some hardware |
| Q3_K_L / Q3_K_M / Q3_K_S | Lower RAM, noticeable quality drop |
| Q2_K | Smallest, most compressed, quality degrades |
Run with llama.cpp, Ollama, LM Studio, or any GGUF-compatible runtime:
./llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_K_M.gguf -p "Your prompt here"
llama.cpp's conversion and quantization tools.