NVIDIA-Nemotron-3-Ultra-550B-A55B-MLX-8bit

NVIDIA-Nemotron-3-Ultra-550B-A55B-MLX-8bit

Other quantizations: 4-bit · 5-bit · 6-bit · 8-bit (this)

📚 Part of the Nemotron-3 MLX (Apple Silicon) collection.

8-bit (MLX) quantization of NVIDIA's Nemotron-3 Ultra 550B-A55B. Converted with mlx-lm from the official BF16 release.

Distributed-only. At ~584 GB this does not fit a single 512 GB Mac — it targets a multi-machine MLX shard (e.g. 512 GB + 256 GB M3 Ultras over Thunderbolt 5, 768 GB combined). For a single 512 GB machine use the 4-bit.

Architecture

Hybrid Mamba-2 + Attention + Latent-MoE (nemotron_h): 108 layers (48 Mamba-2, 48 MoE, 12 Attention), 512 routed experts top-22 + 1 shared, moe_latent_size=2048.

Usage (distributed)

Run sharded across machines with mlx.distributed pipeline/tensor parallelism. See mlx-lm's distributed docs; the model path must exist on every node. mlx-lm >= 0.31.2.

Quantization

• MLX affine 8-bit, group 64. Source: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16.
• Command: mlx_lm.convert --hf-path <bf16> --mlx-path <out> -q --q-bits 8 --q-group-size 64
• Note: the BF16 config omits num_hidden_layers (required by mlx-lm's nemotron_h loader; = 108, from layers_block_type). Inject it into config.json before converting.

Quality note

On the 550B, 4/5/6/8-bit are quality-indistinguishable in our testing (all within noise). 8-bit is provided for max-precision / distributed use, not as a quality upgrade over 4-bit.

Known limitations

• No MTP / speculative decoding (mlx-lm drops the mtp.* layers).
• Context: config sets 256K positions; long contexts grow KV-cache memory.

License

Derived from NVIDIA Nemotron-3 Ultra under OpenMDW-1.1 (included as LICENSE). "Nemotron" is a trademark of NVIDIA. Converted with mlx-lm 0.31.2. Not affiliated with NVIDIA.

<!-- eval-results-v1 -->

Measured eval results

Internal relative-quality harness (n=60, zero-shot length-normalized continuation scoring — not leaderboard-comparable; this method underrates reasoning models vs. their official scores). BF16 reference not measurable (won't fit in memory).

4-bit and 6-bit are within noise on both perplexity and ARC-Challenge — confirming on a real task that 4-bit is quality-equivalent at the 550B scale. (5-bit and 8-bit were not separately ARC-tested; quant-vs-quant perplexity placed all four within ~2%.)