Downloads · 30 days
413
15% of all-time downloads
MRockatansky/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-MTP-GGUF
NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-MTP-GGUF is a text generation model from MRockatansky. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as other.
GGUFs made with a non-mainline build of llama.cpp: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 (official), converted with converthftogguf.py at Premshay/llama.cpp@731368b5d (branch nemotron-super-mtp, stacked on…
Downloads · 30 days
413
15% of all-time downloads
All-time downloads
2.8K
Public
Repo size
328 GB
Likes
3
Public
Click a slice to open those files.
.gguf328 GB · 100%
From the Hugging Face model README
GGUFs made with a non-mainline build of llama.cpp: nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-BF16 (official), converted with convert_hf_to_gguf.py at Premshay/llama.cpp@731368b5d (branch nemotron-super-mtp, stacked on PR #25444), quantized with llama-quantize from the same build
The GGUFs should be loaded with the following fork of llama.cpp on the nemotron-super-mtp branch: https://github.com/Premshay/llama.cpp/tree/nemotron-super-mtp
Just an experiment to try the forked llama.cpp built on top of PR#25444 to support this model's built in MTP heads. Use the fork of llama.cpp above with --spec-type draft-mtp enabled in llama-server to provide a nice speedup in token generation.
The contributor of the fork explained their methodology on the model card below. I just did the same but with Nemotron Puzzle BF16 weights instead of using Nemotron Super: https://huggingface.co/Premshay/Nemotron-3-Super-120B-A12B-MTP-GGUF