Downloads · 30 days
394
24% of all-time downloads
nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15
Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15 is a text generation model from nota-ai. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
On top of NVFP4 quantization, this variant applies additional expert pruning. Rather than removing the same number of experts from every block, Nota AI's proprietary global importance-estimation technique measures exp…
Downloads · 30 days
394
24% of all-time downloads
All-time downloads
1.6K
Public
Parameters
15.5B
36.1 GB on disk
Likes
23
Public
Click a slice to open those files.
.safetensors19 GB · 100%
How the weights are stored.
U812.8B · 82%
From the Hugging Face model README
On top of NVFP4 quantization, this variant applies additional expert pruning. Rather than removing the same number of experts from every block, Nota AI's proprietary global importance-estimation technique measures expert importance across the whole network and keeps only the most important experts on a per-layer basis (global pruning). The number of routed experts therefore varies from layer to layer, recovering accuracy that uniform pruning gives up, while freeing 39% more KV cache on a single 32 GB GPU.
Method clarification. This model does not use uniform expert pruning, in which the same number or proportion of experts is removed from every layer. Instead, we use our proprietary global-scale expert importance score to compare experts across the entire network and determine a different number of retained experts for each layer. Consequently, the pruned architecture has layer-wise variable expert counts, which are explicitly recorded in the model configuration and require the patched model definition provided in this repository.
| NVFP4 | Ours | |
|---|---|---|
| MoE layers | 23 | 23 |
| routed experts per layer | 128 (uniform) | 96 / 100 / 116 (per layer) |
| total routed experts | 2,944 | 2,484 (−15.6%) |
| experts per token (top-k) | 6 | 6 |
| shared experts | 1 | 1 |
| MTP block | 128 experts | 128 experts (unpruned) |
| safetensors on disk | 20.08 GiB | 17.67 GiB |
Capacity is concentrated at the two ends of the stack: the first and last six MoE layers keep 116 experts each, and the eleven middle layers carry the whole reduction at 96–100.
Only MoE experts are pruned. The 6 attention layers, 23 Mamba-2 layers, shared experts and the MTP block are untouched, so the 1M-token context window is unchanged.
Every layer's expert count is a multiple of 4, so tensor/expert parallel sizes of
2 and 4 divide evenly. Counts live in config.json as
n_routed_experts_per_layer; n_routed_experts stays at 128 — the largest count
in the model, held by the unpruned MTP block — so weight mapping still covers
every expert.
Quantization is inherited from the base checkpoint and not modified:
W4A16_NVFP4 with group_size=16 for the experts (4-bit weights, bf16
activations) and FP8 for a small set of Mamba projections, in NVIDIA ModelOpt
format.
Greedy decoding (temperature 0), context length 139,264 tokens, generation
budget max_tokens 131,072, full benchmark sets.
Greedy decoding is used deliberately: it removes sampling variance, so the degradation introduced by compression is measured reproducibly rather than blurred by run-to-run noise.
| benchmark | NVFP4 | REAP (ICLR 2026) | Ours |
|---|---|---|---|
| GPQA-Diamond | 74.24% | 63.64% | 67.68% |
| MMLU-Pro | 80.71% | 76.08% | 76.60% |
| IFEval | 92.05% | 86.88% | 87.25% |
| IFBench | 70.00% | 65.67% | 65.00% |
| HumanEval+ | 87.20% | 84.15% | 87.80% |
| Avg. | 80.84% | 75.28% | 76.87% |
Per-layer allocation lifts the average by +1.58 points over REAP and leads on 4 of the 5 benchmarks; HumanEval+ ends up above the unpruned NVFP4 checkpoint.
Note. REAP is the state of the art for MoE expert pruning and the method behind most pruned MoE checkpoints published on the Hugging Face Hub. The column above is a REAP baseline built from the same saliency scores and the same 15.6% removal rate as ours, removed uniformly from every layer — so the only difference is how the budget is spread across layers.
Measured with vLLM 0.27.1, enforce_eager, serving the full 1M-token context
window.
| on a 32 GB budget | NVFP4 | Ours | change |
|---|---|---|---|
| weights + non-torch | 19.92 GiB | 17.42 GiB | −2.50 GiB |
| peak activation | 2.26 GiB | 2.26 GiB | — |
| KV cache available | 6.43 GiB | 8.93 GiB | +39% |
| KV cache capacity | 2,199,552 tokens | 3,059,712 tokens | +39% |
| concurrent 1M-token requests | 2.10× | 2.92× | +39% |
Both checkpoints start on a single 32 GB card. Pruning does not change whether the model fits — it changes how much of the card is left for serving.
Two things worth knowing:
--kv-cache-dtype fp8 does not enlarge the KV pool here. vLLM pads the
attention page up to the Mamba state page (attention block size 4176 tokens to ensure that attention page size is >= mamba page size), so the Mamba side sets
the floor. At an identical budget, auto and fp8 both give 10.28 GiB /
3,522,560 tokens / 3.36× concurrency on this checkpoint.export MODEL_CKPT=nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15
export VLLM_PATH=/path/to/vllm
vLLM already calls a get_nemotron_h_config_for_layer hook for models with
per-layer expert counts, but the transformers config class that must answer that
hook does not implement it. Copy one file over the installed one:
cp patch/configuration_nemotron_h.py \
$($VLLM_PATH/bin/python -c "import transformers.models.nemotron_h.configuration_nemotron_h as m; print(m.__file__)")
Without it, loading fails with
AssertionError: Attempted to load weight (torch.Size([116])) into parameter (torch.Size([128])),
because every MoE layer would be built with the maximum expert count instead of
its own. Re-apply after any pip install -U transformers.
The deployment recipes from the base model card
carry over unchanged — pruning removes experts but leaves the architecture,
tokenizer, chat template and MTP block intact. vLLM v0.27.1 or newer.
Max throughput, single GPU:
vllm serve --model $MODEL_CKPT \
--max-num-seqs 256 \
--max-num-batched-tokens 16384 \
--enable-prefix-caching \
--async-scheduling \
--mamba-backend flashinfer \
--mamba-cache-mode align \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice
Interactive, low concurrency, with the DSpark drafter for speculative
decoding (export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark):
vllm serve --model $MODEL_CKPT \
--max-num-seqs 128 \
--enable-prefix-caching \
--async-scheduling \
--speculative_config.model $DSPARK_CKPT \
--speculative_config.num_speculative_tokens 3 \
--mamba-backend flashinfer \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice
Multi-GPU (TP4 with expert parallelism):
vllm serve --model $MODEL_CKPT \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--enable-prefix-caching \
--async-scheduling \
--mamba-backend flashinfer \
--mamba-cache-mode align \
--reasoning-parser nemotron_v3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice
Expert parallelism is limited to 2 or 4 ranks.
--enable-expert-parallelsplits each layer's experts across ranks, and this model's per-layer counts are 96, 100 and 116 — divisible by 4 but not by 8. Tensor parallelism without--enable-expert-parallelshards inside each expert instead and is unaffected, so TP8 remains available that way.
Context length. These snippets serve the full 1M-token context window by
default. If you are memory-constrained, or want more KV-cache headroom at high
concurrency, lower --max-model-len to match your workload.
Only plain single-GPU serving was re-validated on this checkpoint; the speculative-decoding, multi-GPU and W4A16/Ampere paths are inherited from the base model card and were not re-tested here.