NVIDIA-Nemotron-3.5-Lightning-30B-A3B-INT4-AutoRound

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-INT4-AutoRound

INT4 (W4A16) AutoRound quantization of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, packed with group_size=64, symmetric — loadable and served directly by vLLM with --quantization auto_round.

Model (unchanged from source)

• Architecture: NemotronHForCausalLM — hybrid Mamba-2 / MoE / attention
• 31.5B total params, ~3.3B active per token (A3B)
• 52 layers: 23 Mamba-2 + 23 MoE + 6 attention
• MoE: 128 routed experts + 1 shared expert, 6 experts/token, expert intermediate 1856, shared-expert intermediate 3712
• Mamba-2: 64 heads × 64 head dim, ssm_state_size 128, conv_kernel 4, chunk_size 128
• Attention: 32 heads, 2 KV heads, head_dim 128
• hidden_size 2688, vocab 131072, designed context 262144

Quantization config

• Tool: auto-round 0.14.2
• bits: 4, group_size: 64, sym (asymmetric off)
• Output format: auto_round (WOQ packing) — vLLM-loadable
• Calibration: 128 samples @ seqlen 512, NeelNanda/pile-10k
• Runtime patches required for auto-round 0.14.2 on this arch: Zamba2RMSNormGated.forward norm patch + 2-projection (NemotronH) expert container patch (_ExpertContainer gate_proj)
• Result: 264 weights packed → 792 packed tensors; 798 tensors not quantized (norms, embeddings, mamba projections, moe router/scaler) → model_extra_tensors.safetensors
• Weights: ~18.5 GB (5 shards + extra tensors)

Why group_size 64 (not 128)

vLLM's MoeWNA16Method halves the effective group size when the MoE expert intermediate dimension is not divisible by 128. This model's expert intermediate is 1856: 1856 % 128 = 64, so vLLM requires g64-grouped weights (1856 / 64 = 29 group rows). A g128 export does not match that layout and fails to load. g64 is the only group size that round-trips cleanly.

Files

• model-00001..05-of-00005.safetensors — quantized weight shards
• model_extra_tensors.safetensors — non-quantized tensors (798)
• model.safetensors.index.json, config.json, quantization_config.json
• tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json

Serving

Requires vLLM 0.27.2rc1.dev18 (nightly) — this exact build is the minimum. The Mamba-2 layers require a GPU mamba backend — flashinfer is the validated choice.

vllm serve letechlead/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-INT4-AutoRound \
  --quantization auto_round \
  --mamba-backend flashinfer \
  --mamba-ssm-cache-dtype float16 \
  --tensor-parallel-size <TP> \
  --gpu-memory-utilization 0.95 \
  --max-model-len <fits your memory> \
  --async-scheduling \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser step3p5

Notes:

• --reasoning-parser nemotron_v3 moves the model's <think> blocks into the reasoning field so content stays clean. This is a reasoning model — it burns tokens thinking before answering, so set generous max_tokens (512+); low budgets yield empty content.
• Tool calling: the model's template emits the NVIDIA XML format (<tool_call><function=...><parameter=...>). The vLLM parser matching it is step3p5 (same format family) — verified working end-to-end.
• --enable-expert-parallel is required for TP >= 4 (expert intermediate 1856/TP not divisible by 32); TP 1-2 work without it, TP=3 unsupported (32 heads).

Context sizing — this is a Mamba-2 model, so context is bounded by the SSM state cache, not the int4 weights. The state cache is activations (~180 KB/token across the 23 mamba layers, fp16) and grows linearly with context; quantization does not shrink it:

Pick --max-model-len to fit your accelerator memory (weights + SSM state + KV cache for the 6 attention layers + workspace). Recommended sampling: temperature 1.0, top_p 0.95 (matches the source model's generation config).

License

Derived from NVIDIA's model — Open Model Development License v1.1 (openmdw-1.1), see the source model card and the license text.