Downloads · 30 days
170
20% of all-time downloads
AlexCaswen/Nemotron-3-Super-120B-A12B-MTP-GGUF
Nemotron-3-Super-120B-A12B-MTP-GGUF is a text generation model from AlexCaswen. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as other.
GGUF quantization of NVIDIA Nemotron-3-Super-120B-A12B-BF16 with the multi-token-prediction (MTP) layers retained, for speculative decoding.
Downloads · 30 days
170
20% of all-time downloads
All-time downloads
851
Public
Repo size
167 GB
Likes
1
Public
Click a slice to open those files.
.gguf78.1 GB · 100%
From the Hugging Face model README
GGUF quantization of NVIDIA Nemotron-3-Super-120B-A12B-BF16 with the multi-token-prediction (MTP) layers retained, for speculative decoding.
Stock llama.cpp cannot load these files. Both conversion and runtime support come from PR #26725.
2f778a3129854a5664ecd813088355e50912bfd2Spec-decode flags: --spec-type draft-mtp --spec-draft-n-max N.
nextn_predict_layers = 1, block_count = 89. All four nextn tensors
(eh_proj, enorm, hnorm, shared_head_norm) sit on blk.88.
Earlier converter revisions emitted a split layout — nextn_predict_layers = 2,
block_count = 90, with shared_head_norm on blk.89. Tensor count is 781
either way. Files built against the split layout are not expected to load on
2f778a31; this file matches the consolidated layout that commit expects.
ffn_down_exps is overriddenexpert_feed_forward_length is 2688, and 2688 / 256 = 10.5. The k-quant
super-block is 256, so k-quants cannot apply to any tensor whose quantized row
length is 2688. ffn_down_exps is [2688, 1024, 512] and hits this; llama.cpp
silently falls back to block-32 legacy types (Q8_0/Q5_0). Its transpose,
ffn_up_exps at [1024, 2688, 512], has row length 1024 and quantizes to Q4_K
as intended.
Those 41 tensors are 57.8B parameters — roughly half the model — so the fallback is expensive:
| build | ffn_down_exps | size | BPW |
|---|---|---|---|
blanket Q4_K_M | Q8_0 / Q5_0 | 82.35 GiB | 5.72 |
| this file | IQ4_NL | 65.28 GiB | 4.54 |
Produced with:
llama-quantize --tensor-type ffn_down_exps=iq4_nl <bf16>.gguf <out>.gguf Q4_K_M
IQ4_NL is 4.5 bpw at block size 32, sidestepping the 256 constraint at
materially better quality than Q4_0. ffn_up_exps keeps Q4_K super-block
scales where the dimensions permit them.
Tensor mix: Q4_K 297 (33.91 GiB) · IQ4_NL 41 (30.27 GiB) · Q6_K 27 (0.77 GiB) · F32 415 (0.32 GiB) · BF16 1.
No imatrix. Straight round-to-nearest.
89 blocks: 41 LatentMoE, 9 global attention, remainder Mamba-2. 512 routed
experts, top-22, plus 1 shared. moe_latent_size 1024. SSM state 128, inner
8192, conv kernel 4, 8 groups. Context 1M. Vocab 131072.
Note for anyone planning VRAM: --n-cpu-moe N indexes layers, not expert
blocks, and only 41 of 89 carry experts — so most values of N free nothing.
MoE layer indices: 1, 3, 5, 8, 10, 12, 14, 17, 19, 21, 23, 26, 28, 30, 32, 34,
37, 39, 41, 43, 45, 48, 50, 52, 54, 56, 59, 61, 63, 65, 67, 70, 72, 74, 76, 79,
81, 83, 85, 87, 88.
| file | notes |
|---|---|
Nemotron-3-Super-120B-A12B-MTP-IQ4NL.gguf | main weights, MTP embedded |
mtp-Nemotron-3-Super-120B-A12B-BF16.gguf | detached draft, for the separate-draft path |
Inherits the NVIDIA Open Model License from the base model. Quantization is a mechanical transformation; no weights were trained or modified beyond precision reduction.