Downloads · 30 days
98
17% of all-time downloads
soyrsoyr/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP is a text generation model from soyrsoyr. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This checkpoint was compressed from nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with LLM Compressor. The backbone follows NVIDIA's hybrid quantization boundary: Mamba input/output projections use calibrated FP8,…
Downloads · 30 days
98
17% of all-time downloads
All-time downloads
591
Public
Parameters
16.7B
38.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors19.6 GB · 100%
How the weights are stored.
U815.3B · 92%
From the Hugging Face model README
This checkpoint was compressed from
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 with LLM Compressor. The
backbone follows NVIDIA's hybrid quantization boundary: Mamba input/output
projections use calibrated FP8, while MoE projections and lm_head use
weight-only NVFP4. Both physical MTP blocks, attention followed by MoE, are
included and their eligible weights are packed to weight-only NVFP4 by the same
oneshot call.
from compressed_tensors.quantization import QuantizationScheme
from compressed_tensors.quantization.quant_scheme import FP8, NVFP4
from transformers import AutoModelForCausalLM
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.utils import load_context
model_id = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"
recipe = QuantizationModifier(
config_groups={
"mamba": QuantizationScheme(
targets=[
r"re:^model\.layers\.\d+\.mixer\.(?:in_proj|out_proj)$"
],
**FP8,
),
"moe": QuantizationScheme(
targets=[
r"re:^model\.layers\.\d+\.mixer\."
r"(?:experts\.\d+|shared_experts)\."
r"(?:gate_proj|up_proj|down_proj)$",
"lm_head",
],
weights=NVFP4["weights"],
),
},
ignore=[
"model.embed_tokens",
r"re:^model\.layers\.\d+\.mixer\.conv1d$",
r"re:^model\.layers\.\d+\.mixer\.gate$",
],
)
with load_context(AutoModelForCausalLM):
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")
oneshot(
model=model,
recipe=recipe,
dataset="perfectblend",
splits="train[:512]",
output_dir="/workspace/models/Nemotron-3.5-Lightning-30B-NVFP4-MTP",
max_seq_length=2048,
num_calibration_samples=512,
batch_size=1,
sequential_targets=["NemotronHBlock"],
mtp_scheme="NVFP4",
)
Transformers does not construct Nemotron's MTP modules when loading the causal LM. LLM Compressor therefore reads those tensors from the source checkpoint during oneshot post-processing, packs eligible MTP weights, writes a dedicated MTP shard, and updates the safetensors index and compressed-tensors config. Static MTP activation scales cannot be calibrated without executable MTP modules, so the MTP group is weight-only NVFP4. The loaded backbone retains its calibrated hybrid configuration.
model_mtp.safetensors; no backbone
tensors are present in that shard.mtp.layers.0 is full attention and
mtp.layers.1 is MoE.vllm serve soyrsoyr/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP \
--tensor-parallel-size 2 \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
curl -sS -X POST http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "soyrsoyr/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP",
"messages": [{"role": "user", "content": "Write a Python function that returns the nth Fibonacci number, then explain how it works."}],
"max_tokens": 256,
"temperature": 0
}'