Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant-MLX-MXFP4

[!TIP] KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1).

The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

<!-- kv-upstream-note -->

Nemotron-3-Nano-Omni-30B-A3B-Reasoning - RotorQuant MLX MXFP4

MLX-native MXFP4 quantization of the text tower of Nemotron-3-Nano-Omni-30B-A3B-Reasoning (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) with RotorQuant weight method. Apple Silicon native.

This is an out-of-matrix variant — separate from the standard MLX int-bit ladder (2/3/4/5/6/8-bit), parallel to the Qwen 3.6 MLX-MXFP4 precedent. Text tower only, same caveat as the int-bit MLX variants regarding multimodal encoders.

For the runtime KV-cache modifier itself, see majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant.

Quickstart

# Today (mlx-lm 0.31.x): the NemotronH_Nano_Omni_Reasoning_V3 model class
# is not yet registered in mlx-lm. The cell below is the API shape that WILL
# work once upstream lands the class (track ml-explore/mlx-lm#386).

from mlx_lm import load, generate

model, tokenizer = load("majentik/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-RotorQuant-MLX-MXFP4")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Solve: 17 * 23"}],
    add_generation_prompt=True,
    enable_thinking=False,  # set True to enable extended reasoning (default)
)

response = generate(
    model, tokenizer,
    prompt=prompt,
    max_tokens=512,
    sampler=lambda x: x.argmax(axis=-1),  # or use mlx_lm.sample_utils.make_sampler(temp=0.6, top_p=0.95)
)
print(response)

⚠️ This variant covers the text tower only. For multimodal inference (vision + audio + video), use the GGUF variants with llama-mtmd-cli — see the GGUF cards in this family.

MXFP4 microscaling format. This variant uses MXFP4 (E2M1 mantissa, shared E8M0 scale) with group_size=32 and no per-group bias — distinct from the GGUF MXFP4_MOE packing. Loadable by mlx-lm once the NemotronH model class lands upstream.

Modality matrix

NVIDIA's official FP8 / NVFP4 recipe keeps both encoders + the cross-modal MLP projectors in BF16 to preserve multimodal accuracy. We follow that convention in every quantized variant we ship.

Runtime quirks

vLLM / transformers

Standalone MXFP4 (not the MXFP4_MOE-in-GGUF format). Loadable by vLLM ≥ 0.6.x and transformers + auto-awq backend. Encoders remain BF16.

Required vLLM flags:

--reasoning-parser nemotron_v3
--media-io-kwargs '{"video":{"fps":2,"num_frames":256}}'

Reasoning mode

enable_thinking defaults to True. To disable extended reasoning (e.g., for latency-sensitive cases), pass enable_thinking=False to the chat template / generate call. No separate "no-think" variant card exists — this is a runtime flag, not a model variant.

Variants in this family

(Showing 56 sibling variants under majentik/nemotron3-nano-omni-30b-*. The current variant — RotorQuant-MLX-MXFP4 — is bolded.)

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured.