Downloads · 30 days
41
12% of all-time downloads
josefprusa/Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP
Nemotron-Labs-3-Puzzle-75B-A9B-oQNVFP4-MTP is a text generation model from josefprusa. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as other.
Experimental, AI-built — not production software. This artifact and its serving stack were built largely by AI agents under human direction and validated by measurement, not review. Enjoy the speed, hack on it, review…
Downloads · 30 days
41
12% of all-time downloads
All-time downloads
339
Public
Parameters
75.4B
51.9 GB on disk
Likes
2
Public
Click a slice to open those files.
.safetensors51.8 GB · 100%
How the weights are stored.
U3275.3B · 100%
From the Hugging Face model README
Experimental, AI-built — not production software. This artifact and its serving stack were built largely by AI agents under human direction and validated by measurement, not review. Enjoy the speed, hack on it, review it before you depend on it — and if you'd like to adopt and polish it properly, please do; the author would be delighted and simply doesn't have the time.
NVIDIA's Nemotron-3 Puzzle 75B (A9B active, hybrid Mamba/attention/MoE) quantized to
oQNVFP4 (NVFP4 MoE weights with per-layer time-scale sidecars, 6-bit affine elsewhere,
~48 GB) for Apple Silicon, with the checkpoint's multi-token-prediction head included
as an mtp.safetensors sidecar (5.5 GB, bf16, bit-exact extraction of NVIDIA's
"training-only" MTP tensors).
Most community conversions drop the MTP tensors. Kept and served, they roughly halve decode latency: the head drafts multi-token chains that the trunk verifies in a single fused pass, and acceptance on natural text is high enough (~80–86%) that speculation pays at every context depth.
Apple M3 Ultra 512 GB, omlx fork, streaming API, salted natural-document prompts (no repeated-filler inflation), temperature 0, 1000-token generations at depth:
| context | decode tok/s (MTP on) | decode tok/s (MTP off) | prefill tok/s |
|---|---|---|---|
| 0.6k | 85.0 | 55.0 | — |
| 2k | 80.8 | 54.8 | 977 |
| 26k | 63.0 | — | 977 |
| 53k | 57.3 | — | 897 |
| 108k | 47.5 | — | 771 |
MTP census over full runs: 2.99 tok/cycle at short context, ~1.86 at 26k+ (the adaptive depth controller shallows as verify forwards grow), acceptance stable at 80–86% throughout. Prefill is near-flat with depth — the Mamba backbone's signature.
This artifact is not loadable by stock mlx-lm. The NVFP4 time-scale sidecars
(fc1_ts/fc2_ts) and the MTP sidecar head require the
omlx fork:
git clone https://github.com/josefprusa/omlx
cd omlx && uv sync --python 3.12
OMLX_WITH_CUSTOM_KERNEL=1 uv pip install -e . # builds the native kernels (needs full Xcode)
omlx serve
Place this repository's contents in your omlx models directory, enable mtp_enabled for
the model in model_settings.json, and the server logs will show
[Nemotron-H MTP] sidecar head attached (pattern=*E, depth=3) plus a per-request
acceptance census.
Works without MTP too (55 tok/s class): leave mtp_enabled off and the model serves as a
plain checkpoint — the sidecar is simply ignored.
mtp.* block (bf16,
unquantized) — provenance-verifiable by hash against the source shards.experimental/.License: OpenMDW v1.1 (NVIDIA's open model license), carried unchanged in LICENSE.