NVIDIA-Nemotron-3.5-Lightning-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B (GGUF)

GGUF conversion of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, an open 30B mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. Converted for use with llama.cpp using imatrix-calibrated quantization.

One quantization is provided:

Model Summary

Files

impacte/NVIDIA-Nemotron-3.5-Lightning-GGUF/
├── NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf   # IQ4_XS imatrix GGUF (18.7 GB)
└── .gitattributes                                        # LFS tracking

Usage

llama.cpp (local inference)

# IQ4_XS (24 GB GPU, full 256K context)
llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
  --ctx-size 262144 \
  --port 8080

For the full 256K context, use a quantized KV cache (--cache-type-k q4_0 --cache-type-v q4_0) to keep the KV cache footprint minimal:

llama-server \
  -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf \
  --ctx-size 262144 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --port 8080

Then call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS",
    "messages": [
      {"role": "user", "content": "Explain what a Tauri v2 app is."}
    ]
  }'

Ollama

# Pull the pre-built Ollama tag (256K context, q4_0 KV cache)
ollama run oamazonasgabriel/nemotron-3.5-lightning:iq4-xs-256k-24gbGPU

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-IQ4_XS.gguf",
    n_ctx=262144,
    n_gpu_layers=-1,  # offload all layers to GPU
)

About the base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B is an open 30B-parameter mixture-of-experts (MoE) model with ~3B active parameters, built by NVIDIA for the execution layer of always-on agents. It uses a hybrid Mamba-2 + MoE + attention architecture with only ~6 attention layers out of 52, so the KV cache stays tiny even at very long contexts. The model supports native tool calling and thinking mode, and is multilingual (en, es, fr, de, it, ja).

License & Attribution

• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 — released under the OpenMDW-1.1 license. Review before commercial use.
• GGUF conversion: performed with llama.cpp's convert_hf_to_gguf.py (bf16) and llama-quantize (IQ4_XS with imatrix calibration).

Note: This is a GGUF conversion of a model under the OpenMDW-1.1 license. Ensure your use complies with that license.

Limitations

• The IQ4_XS GGUF (18.7 GB) fits on a 24 GB GPU (16 GB + 8 GB dual) with room for a 256K KV cache at q4_0, but trades some precision vs higher quants (Q6_K / Q8_0).
• The base model's general capabilities are retained; this is a direct conversion with no additional fine-tuning.