Nemotron-3.5-Lightning-30B-Q4_K_M-Imatrix

Nemotron-3.5-Lightning-30B-A3B — Q4_K_M GGUF (imatrix)

A Q4_K_M GGUF quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, built with a custom importance matrix.

Every model figure on this card comes from the GGUF file itself or the base model config, so you can check it.

---

Why imatrix

An importance matrix records which weights actually matter for the model's predictions, so the quantizer spends its limited precision where it counts. This one was computed over 604 chunks of Wikitext-2 (train) and is embedded in the GGUF metadata (quantize.imatrix.*).

---

Hardware requirements

The weights alone are 23.68 GiB, so a 24 GB card is not enough. Context is cheap on top of that: only 6 of the 52 layers are attention layers with a growing KV cache (2 KV heads × 128 dims, 6 KiB per token at F16), and the 23 Mamba-2 layers keep a fixed ~48 MiB state.

Computed from the GGUF metadata. Add roughly 1–2 GiB for the CUDA context and compute buffers, depending on batch size. For 1M context on a 32 GB card, use --cache-type-k q8_0 --cache-type-v q8_0 (needs flash attention) to halve the KV cache.

Each generated token reads only about 2.32 GiB of these weights — the 6 routed experts picked out of 128 plus the shared layers, a tenth of the file.

---

Run it on Meshive

Meshive is a task-first GPU cloud. Bring your task — training, inference, or rendering — and run it on RTX 5090 and RTX PRO GPUs at up to 70% less.

RTX PRO 6000 Blackwell and 32 GB RTX 5090 are both available as on-demand GPU Pods on Meshive.

---

Usage

llama.cpp

Make sure you are using a build with native Nemotron MoE kernel support.

llama-cli -m Nemotron-3.5-Lightning-30B-Q4_K_M-Imatrix.gguf -ngl 999 -c 32768 \
  -p "Explain the concept of quantum computing in simple terms."

As an OpenAI-compatible server:

llama-server -m Nemotron-3.5-Lightning-30B-Q4_K_M-Imatrix.gguf -ngl 999 -c 32768 --host 0.0.0.0 --port 8080

Ollama

printf 'FROM ./Nemotron-3.5-Lightning-30B-Q4_K_M-Imatrix.gguf\n' > Modelfile
ollama create nemotron-3.5-imatrix -f Modelfile
ollama run nemotron-3.5-imatrix

Chat template

The GGUF contains the embedded chat template from the base model. llama.cpp and Ollama apply it automatically. Avoid forcing legacy templates to preserve proper tool calling and thinking/reasoning delimiters.

---

Quantization method

Base model     nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16  →  F16 GGUF via convert_hf_to_gguf.py
Quantizer      llama.cpp llama-quantize, Q4_K_M
Imatrix        Wikitext-2 (train), computed over F16
Hardware       NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB)
Notes          142 of 173 quantized tensors fall back to Q5_0 / Q8_0 (see below)

Most of this file is not Q4_K. K-quants need rows that are a multiple of 256 values, and this model's hidden size (2688) and expert width (1856) are not. llama.cpp therefore falls back to Q5_0 where Q4_K_M asks for Q4_K, and to Q8_0 where it asks for Q6_K. By size the file is 68% Q5_0, 31% Q8_0 and under 1% Q4_K — about 6.2 bits per weight, larger than the Q4_K_M name suggests. The MoE router weights are not quantized.

Measuring quality against BF16

# 1. BF16 reference logits
llama-perplexity -m base-f16.gguf -f wiki.test.raw -c 512 -b 512 -ngl 999 \
  --kl-divergence-base ref.kld

# 2. Score this model against them
llama-perplexity -m Nemotron-3.5-Lightning-30B-Q4_K_M-Imatrix.gguf -f wiki.test.raw \
  -c 512 -b 512 -ngl 999 --kl-divergence-base ref.kld --kl-divergence

---

Credits

Quantized by meshive. Base model by NVIDIA, released under the OpenMDW License Agreement 1.1 — see LICENSE for the full terms and NVIDIA's copyright notice.