NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF

NVIDIA-Nemotron-3-Ultra-550B-A55B-GGUF

GGUF quantized version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.

How can I use?

1. Upstream llama.cpp: Model probably works.
2. Forked version of llama.cpp: RPC with split mode tensor, MTP, and model works

Detailed information

• Model: 161667.64 MiB (2.47 BPW)
• MTP: 8882.72 MiB (5.58 BPW)

How it was made?

Model

export TYPE_FFN_UP_EXPS=IQ2_XXS
export TYPE_FFN_DOWN_EXPS=IQ2_XS

export TYPE_LATENT=Q8_0
export TYPE_SHEXP=Q6_K
export TYPE_SSM=Q6_K
export TYPE_ATTN=Q6_K

export TYPE_DEFAULT=Q4_K_M

llama-quantize \
    --tensor-type 'ffn_up_exps='$TYPE_FFN_UP_EXPS \
    --tensor-type 'ffn_down_exps='$TYPE_FFN_DOWN_EXPS \
    --tensor-type 'ffn_latent_(up|down)='$TYPE_LATENT \
    --tensor-type 'ffn_(up|down)_shexp='$TYPE_SHEXP \
    --tensor-type 'ssm_(in|out)='$TYPE_SSM \
    --tensor-type 'attn_(q|k|v|output)='$TYPE_ATTN \
    --imatrix imatrix.gguf \
    "model-BF16.gguf" "$TYPE_DEFAULT"

MTP

    llama-quantize \
    --tensor-type 'nextn\.eh_proj=Q6_K' \
    --tensor-type 'ffn_latent_(up|down)=Q8_0' \
    --tensor-type 'ffn_(up|down)_shexp=Q6_K' \
    --tensor-type 'attn_(q|k|v|output)=Q6_K' \
    "mtp-BF16.gguf" "mtp-quant.gguf" Q4_K_M

imatrix

• Generate Korean-focused imatrix calibration dataset (Total 512 samples, Chat template applied for messages)

• Create imatrix using llama-imatrix with --parse-special flag

Benchmark

Where is PPL/KLD?

Unfortunately, I don't have PPL/KLD for quant, due to out of compute resource and target budget.

Throughtput

2x DGX Sparks, RPC, -sm tensor with MTP enabled (-ub 2048 -b 2048 -c 262144)