NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit

Built with mlx-optiq, the MLX-native toolkit to quantize, fine-tune, and serve LLMs locally on Apple Silicon, no PyTorch and no cloud. All OptiQ quants · Docs

NVIDIA's Mamba-attention hybrid MoE, running locally on a Mac. This is an OptiQ mixed-precision quant of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16. 22.8 GB on disk, down from 65.8 GB.

Lightning is a nemotron_h hybrid: Mamba2 state-space blocks with a handful of attention layers, and a 128-expert MoE with 6 experts active per token. The routed experts carry most of the weight, so that is where the bits come from.

What it is

All 164 quantizable projections were measured. Every 4-bit assignment in the model is an expert tensor: the 128-expert stack is where the size is, and the sweep found it is also where the precision is least needed.

Following llama.cpp's naming for mixed quants, the "4bit" label denotes the family, not the weighted average.

Capability Score

Six-metric mean (the standard OptiQ text eval). Strong code generation; long-context multi-hop retrieval is the weak spot, which is what a hybrid with few attention layers would predict.

Run it

pip install "mlx-optiq>=0.4.20"

import optiq  # registers the nemotron_h MoE + MTP sidecar
from mlx_lm import load, generate

model, tok = load("mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit")
msgs = [{"role": "user", "content": "Explain why the sky is blue."}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
print(generate(model, tok, prompt=prompt, max_tokens=600))

It is a reasoning model and emits a <think> block before its answer, so give it room and read the text after </think>.

Serve it on a smaller Mac

At 22.8 GB this does not fit resident on a 24 GB machine. It is a 128-expert MoE, so stream the experts from SSD instead:

optiq serve --model mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-OptiQ-4bit --stream-experts

Measured on a 26 GB M4 Pro: 4.2 GB resident, 6.0 GB load peak, against a resident load that runs out of memory. The same policy is available from the library API via OptiqEngine(path, stream_experts="auto"), and from optiq eval --stream-experts for benchmarking.

Links

• Project website: mlx-optiq.com
• Family guide: mlx-optiq.com/docs/nemotron3
• All OptiQ quants: mlx-optiq.com/models
• Base model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16