Downloads · 30 days
339
26% of all-time downloads
prithivMLmods/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF is a text generation model from prithivMLmods. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is the full-precision reference release of NVIDIA's Nemotron 3.5 Lightning model, a hybrid Mixture-of-Experts architecture interleaving Mamba-2 and MoE layers with select Att…
Downloads · 30 days
339
26% of all-time downloads
All-time downloads
1.3K
Public
Repo size
153 GB
Likes
3
Public
Click a slice to open those files.
.gguf153 GB · 100%
From the Hugging Face model README
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 is the full-precision reference release of NVIDIA's Nemotron 3.5 Lightning model, a hybrid Mixture-of-Experts architecture interleaving Mamba-2 and MoE layers with select Attention layers, totaling 30B parameters with only 3B active at inference and supporting up to 1M tokens of context (256K on a single H100). Pre-trained on over 20 trillion tokens using an NVFP4 recipe and enhanced with Multi-Token Prediction (MTP) layers for richer training signals, it underwent a four-stage pipeline — pretraining, MTP continued pretraining, supervised fine-tuning on code/math/science/tool-calling data, and multi-environment GRPO reinforcement learning across math, code, instruction-following, and structured-output tasks — supporting configurable reasoning mode (
enable_thinking), English plus five additional languages, and speculative decoding via DSpark, DFlash, or MTP for faster generation. This BF16 checkpoint is intended primarily as a starting point for customization — post-training (SFT/RL/distillation), domain adaptation, or producing quantized (NVFP4, W4A16, GGUF) variants — rather than direct production deployment, for which NVIDIA recommends the companion NVFP4 release; on NVIDIA's internal benchmark harness it trails larger sibling Qwen3.6-35B-A3B and Nemotron 3 Super on most agentic coding and reasoning tasks (e.g., 51.56 on SWE-bench Verified, 75.44 on GPQA Diamond) while remaining competitive with Gemma4-26B-A4B and GPT-OSS-20B, deployable on single or multi-GPU H100/H200/GB200/A100 setups via vLLM, and released under the OpenMDW-1.1 license alongside NVIDIA's full pre-training and post-training dataset lineage.
| File Name | Quant Type | File Size | File Link |
|---|---|---|---|
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.F16.gguf | F16 | 65.9 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf | Q4_K_M | 25.4 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q5_K_M.gguf | Q5_K_M | 27 GB | Download |
| NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q8_0.gguf | Q8_0 | 35 GB | Download |
FROM ghcr.io/ggml-org/llama.cpp:full
WORKDIR /app
RUN apt update && apt install -y python3-pip
RUN pip install -U huggingface_hub --break-system-packages
RUN python3 -c 'from huggingface_hub import hf_hub_download; \
repo="prithivMLmods/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16-GGUF"; \
hf_hub_download(repo_id=repo, filename="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", local_dir="/app")'
CMD ["--server", \
"-m", "/app/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.Q4_K_M.gguf", \
"--host", "0.0.0.0", \
"--port", "7860", \
"-t", "2", \
"--cache-type-k", "q8_0", \
"--cache-type-v", "iq4_nl", \
"-c", "128000", \
"-n", "38912"]
LLM inference in C/C++ — https://github.com/ggml-org/llama.cpp