Downloads · 30 days
276
11% of all-time downloads
WhiskyAKM/Nemotron-3.5-Lightning-30B-A3B-GGUF
Nemotron-3.5-Lightning-30B-A3B-GGUF is a text generation model from WhiskyAKM. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as openmdw-1.1.
This repository contains GGUF format model files for NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B.
Downloads · 30 days
276
11% of all-time downloads
All-time downloads
2.4K
Public
Repo size
256 GB
Likes
0
Public
Click a slice to open those files.
.gguf256 GB · 100%
From the Hugging Face model README
This repository contains GGUF format model files for NVIDIA's NVIDIA-Nemotron-3.5-Lightning-30B-A3B.
These files were converted and quantized using llama.cpp.
| Filename | Quant Method | Description |
|---|---|---|
nemotron-3.5-lightning-30b-a3b-bf16.gguf | BF16 | Full-precision reference weights (unquantized) |
nemotron-3.5-lightning-30b-a3b-Q8_0.gguf | Q8_0 | Extremely high quality, fast, high resource usage |
nemotron-3.5-lightning-30b-a3b-Q6_K.gguf | Q6_K | Very high quality, near-lossless quantization |
nemotron-3.5-lightning-30b-a3b-Q5_K_M.gguf | Q5_K_M | High quality, balanced performance and memory |
nemotron-3.5-lightning-30b-a3b-Q5_K_S.gguf | Q5_K_S | High quality, slightly smaller footprint than Q5_K_M |
nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf | Q4_K_M | Recommended balance of size, speed, and quality |
nemotron-3.5-lightning-30b-a3b-Q4_K_S.gguf | Q4_K_S | 4-bit quantization with small memory footprint |
nemotron-3.5-lightning-30b-a3b-Q4_0.gguf | Q4_0 | Standard 4-bit quantization |
nemotron-3.5-lightning-30b-a3b-dflash.gguf | — | DFlash speculative decoding draft model |
| Total Parameters | 30B (3B active) |
|---|---|
| Architecture | MoE — Mamba-2 + MoE + Attention hybrid |
| Context Length | Up to 1M tokens (256K native default) |
| Supported Languages | English (and coding languages), Spanish, French, German, Italian, Japanese |
| Speculative Decoding | DSpark, DFlash, MTP (Multi-Token Prediction) |
| Reasoning Mode | Configurable on/off via chat template (enable_thinking=True/False) |
| Recommended Sampling | Temperature 1.0, Top_P 0.95 |
| License | OpenMDW License Agreement, version 1.1 |
| Original Model | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 |
Model Developer: NVIDIA Corporation
Model Dates: December 2025 - May 2026
Data Freshness:
NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.
NVIDIA-Nemotron-3.5-Lightning-30B-A3B is a large language model (LLM) trained by NVIDIA.
The model employs a hybrid Mixture-of-Experts architecture, utilizing interleaved Mamba-2 and MoE layers, along with select Attention layers. The Lightning 3.5 model is released alongside speculative decoding methods (DSpark, DFlash, MTP) for faster text generation. The model has 3B active parameters and 30B parameters in total.
This model is ready for commercial use under the OpenMDW-1.1 license.
Reasoning ON (default):
llama-cli \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
--jinja \
--chat-template-file chat_template.jinja \
-p "Write a Python function to compute Fibonacci numbers." \
--temp 1.0 --top-p 0.95 \
-ngl 99
Start the OpenAI-compatible server:
llama-server \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
--temp 1.0 --top-p 0.95 \
-np 1 \
-c 40960 \
--port 8000 \
-ngl 99 \
-fa on \
--jinja \
--chat-template-file chat_template.jinja \
--no-webui \
--fit off
Accelerate token generation using the DFlash draft model:
llama-server \
-m nemotron-3.5-lightning-30b-a3b-Q4_K_M.gguf \
-md nemotron-3.5-lightning-30b-a3b-dflash.gguf \
--draft-max 6 \
--temp 1.0 --top-p 0.95 \
-np 1 \
-c 40960 \
--port 8000 \
-ngl 99 \
-ngld 99 \
-fa on \
--jinja \
--chat-template-file chat_template.jinja \
--no-webui \
--fit off
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
# Reasoning ON (default)
response = client.chat.completions.create(
model="nemotron-3.5-lightning-30b-a3b",
messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
max_tokens=4096,
temperature=1.0,
top_p=0.95,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.content)
# Reasoning OFF (direct answer)
response = client.chat.completions.create(
model="nemotron-3.5-lightning-30b-a3b",
messages=[{"role": "user", "content": "What is the capital of Japan?"}],
max_tokens=128,
temperature=1.0,
top_p=0.95,
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(response.choices[0].message.content)
| Task | Nemotron-3.5-Lightning-30B-A3B-BF16 | Nemotron-3.5-Lightning-30B-A3B-NVFP4 |
|---|---|---|
| General Knowledge | ||
| MMLU Pro | 81.94 | 81.62 |
| AA-Omniscience | 17.50 | 16.63 |
| Reasoning | ||
| GPQA Diamond (no tools) | 75.44 | 75.57 |
| HLE (text-only, no tools) | 11.72 | 10.47 |
| SciCode | 32.60 | 31.38 |
| Coding & Agentic | ||
| SWE-bench Verified | 51.56 | 52.80 |
| SWE-bench Multilingual | 39.33 | 36.47 |
| Terminal-Bench 2.1 | 24.58 | 23.46 |
| PinchBench | 85.37 | 83.43 |
| BrowseComp | 36.97 | 36.81 |
| τ³-bench (Banking) | 9.28 | 9.48 |
| GDPval-AA-V2 | 832 | 865 |
| Instruction Following | ||
| IFBench (loose) | 71.88 | 72.88 |
| Long Context | ||
| AA-LCR | 52.00 | 49.19 |
Governing Download Terms: Use of this model is governed by the OpenMDW-1.1 model license.