Downloads · 30 days
55
5% of all-time downloads
pirola/Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS-GGUF
Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS-GGUF is a text generation model from pirola. Use it when you need the model to write or continue text. The card lists the license as other.
A 14.33 GiB GGUF of NVIDIA-Nemotron-3.5-Lightning-30B-A3B that serves the full 262,144-token context on a single 16 GB GPU — with genuine IQ3XXS expert tensors, which no stock build of this model can produce.
Downloads · 30 days
55
5% of all-time downloads
All-time downloads
1.1K
Public
Repo size
15.4 GB
Likes
2
Public
Click a slice to open those files.
.gguf15.4 GB · 100%
From the Hugging Face model README
A 14.33 GiB GGUF of NVIDIA-Nemotron-3.5-Lightning-30B-A3B that serves the full
262,144-token context on a single 16 GB GPU — with genuine IQ3_XXS expert
tensors, which no stock build of this model can produce.
⚠️ This file requires a patched llama.cpp. Two separate reasons (see Building): the expert tensors are zero-padded, and this model needs upstream Dflash support that only landed on 2026-08-11.
Nemotron-3.5-Lightning's MoE expert tensors are 1856 / 2688 / 3712 wide. None is
divisible by 256, so llama-quantize silently falls back to block-32 types for every
block-256 (K/IQ) quant — the model's file size stops responding to the requested
bit-width.
This is not theoretical. Reading the tensor census straight out of the published
unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/…-UD-IQ3_XXS.gguf (19.76 GB):
| tensor group | actual type |
|---|---|
| 46 trunk expert tensors | IQ4_NL (block-32, ~4.5 bpw) |
blk.52 MTP experts | Q5_0 |
| everything else | Q8_0 ×95, Q6_K ×30, F32 ×243, BF16 ×1 |
| IQ3_XXS tensors | 0 |
A file named IQ3_XXS that contains no IQ3_XXS at all. Every published sub-4-bit GGUF
of this family has the same shape.
The fix: zero-pad the reduction axes so block-256 quants actually apply.
| tensor | original | padded |
|---|---|---|
ffn_up_exps reduction axis | 2688 | 2816 |
ffn_down_exps (expert FF width) | 1856 | 2048 |
| shared-expert FF width | 3712 | 3840 |
This is exact, not approximate: the MoE activation is relu², and relu²(0) = 0,
so the padded rows contribute exactly zero. Zero is exactly representable in every ggml
block format, so the padding stays inert after quantization.
Result: genuine IQ3_XXS experts, and the whole 262k context fits a 16 GB card.
| metric | value |
|---|---|
| File | 15,387,569,504 bytes (14.33 GiB) |
| VRAM @ 262,144 ctx, q4_0 KV, after load | 14,784 MiB |
| VRAM @ full 250k-token prompt | 14,878 MiB (1,100 MiB free) |
| Prefill @ 250k | 3,819 t/s |
| Decode @ 250k depth | 78.8 t/s |
| Generation smoke test | 3/3 pass |
Quality benchmarks (MBPP / HumanEval / agentic) have not been run on this build yet — see Status below. Nothing here claims a quality result.
0ba17723bce5f3e7faaf1941c6a3c393249f379c790fcb55d72e221095821aa8 Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf
Stock llama.cpp cannot load this file. You need current master (for Dflash) plus the padding patch included here.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Must include PR #26905 "Dflash support for nemotron-3.5" (merged 2026-08-11).
# Any master checkout after that date is fine.
git apply /path/to/nemotron-expert-padding.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CUDA_ARCHITECTURES=120 # sm_120 = RTX 5080; use your own GPU's capability
cmake --build build --config Release -j
The patch is 3 files / 21 inserted lines: read the padded width from a GGUF key, size
ffn_up_exps from it (in both the trunk and MTP paths), ggml_pad the MoE input to
match, and size the per-expert output views from the down projection.
Toolchain caveat: NVCC 13.2 miscompiles the i-quant CUDA kernels (
iq1_s/iq2_s/iq3_sfailtest-backend-ops test -o MUL_MAT). This build was made with CUDA 13.0. If you are on 13.2+, rebuild against 13.0 before trusting IQ output.
llama-server \
-m Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf \
--host 127.0.0.1 --port 8189 --alias lightning \
-ngl 99 -np 1 --flash-attn on \
-c 262144 -ctk q4_0 -ctv q4_0 \
--jinja --temp 0.6 --top-p 0.95
serve-lightning.sh wraps this. Sampling, per NVIDIA's card for the family: 0.6 / 0.95
for tool calling (the served default above), 1.0 / 1.0 for reasoning — set
per-request. Thinking is on by default; disable per-request with
"chat_template_kwargs": {"enable_thinking": false}.
Lightning ships a multi-token-prediction head — blk.52, declared via block_count = 53
and nextn_predict_layers = 1. It is a full extra MoE layer (its own 128 experts).
It is retained in this file, with its experts at Q5_0. Two things to know:
mtp_flags = !load_mtp ? TENSOR_SKIP : 0,
and load_mtp is false unless you pass --spec-type draft-mtp. On a normal load the
server logs model has unused tensor blk.52.… -- ignoring for all 16 of them, and
they are never allocated. The 14,878 MiB measured above is with the MTP head in the
file.llama-imatrix only
observes tensors on the normal forward path, and the MTP head isn't on it — so no
importance data exists for blk.52, and llama-quantize correctly refuses
("Missing importance matrix … in a very low-bit quantization … bailing out"). Q5_0 is
a block-32 type that needs none. This is the same approach unsloth shipped.Speculative decoding via --spec-type draft-mtp is available but has not been tested
here. If you try it, results are welcome in the discussions.
Do not try to shrink this by pruning layer 52:
--prune-layers 52updatesblock_countbut not the per-layer array KVs, and the result fails to load withkey nemotron_h_moe.feed_forward_length has wrong array length; expected 52, got 53.
imatrix-padded-lightning.gguf). Unsloth's
own quant metadata references
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/imatrix_unsloth.gguf
(unsloth_calibration_NVIDIA-Nemotron-3.5-Lightning-30B-A3B.txt, 80 chunks, 185
entries), but that file is not present in their repo — they published one for
Nemotron-3-Nano but not for Lightning. We have asked. If it appears, this model will be
rebuilt with it and the two compared directly.| file | purpose |
|---|---|
Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf | the model |
nemotron-expert-padding.patch | required llama.cpp patch (3 files, 21 lines) |
imatrix-padded-lightning.gguf | bartowski's Lightning imatrix, padded to match |
pad_gguf.py | pads a BF16 master (the technique) |
pad_imatrix.py | pads an imatrix to the padded geometry |
verify_pad.py | asserts the padded regions are exactly zero |
serve-lightning.sh | the serve command above |
# 1. pad a BF16 master (e.g. unsloth's 2-shard BF16 GGUF)
python pad_gguf.py --variant b --in <master-*.gguf> --out lightning-padded-bf16.gguf
# 2. pad the imatrix to match
python pad_imatrix.py --variant b --in <imatrix.gguf> --out imatrix-padded-lightning.gguf
# 3. quantize with the patched build.
# blk.52 rules MUST come first -- the first matching --tensor-type rule wins.
llama-quantize --imatrix imatrix-padded-lightning.gguf \
--tensor-type "blk\.52\.ffn_up_exps=q5_0" \
--tensor-type "blk\.52\.ffn_down_exps=q5_0" \
--tensor-type ffn_up_exps=iq3_xxs \
--tensor-type ffn_down_exps=iq3_xxs \
lightning-padded-bf16.gguf out.gguf Q4_K_M 14
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, architecture
nemotron_h_moe (Mamba2 + MoE hybrid, 128 experts, 52 trunk layers + 1 MTP layer,
native 262,144 context).Nemotron-3-Nano-30B-A3B-pirola-IQ3_XXS-GGUF
and the 12 GB variant
Nemotron-3-Nano-30B-A3B-pirola-IQ2_XXS-XS-GGUF.