Nex-N2-mini-NVFP4-GGUF

Nex-N2-mini NVFP4 GGUF

NVFP4 GGUF quantizations of nex-agi/Nex-N2-mini, produced for use with llama.cpp.

This is a MoE model — 35B total parameters, ~3B activated per token (8 of 256 experts). The expert FFN tensors — both routed experts (*_exps) and shared experts (*_shexp), 240 tensors total — are quantized to NVFP4 (NVIDIA's 4-bit float with E4M3 block scales), repacked from the calibrated r0b0tlab/nex-n2-mini-nvfp4 checkpoint (NVIDIA ModelOpt v0.44). Because the experts dominate the model's memory footprint, NVFP4-quantizing them gives most of the size reduction; the remaining tensors (attention, linear-attention blocks, embeddings) use a conventional GGUF quant.

About LibertAI

LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the expert matmul — the dominant compute cost during MoE inference. On older GPUs the path falls back to dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.

Files

Performance

Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build 85f99dca8.

Variant comparison (single-stream, llama-bench 512 in / 64 out)

Batched serving vs stock Q4_K_M (honest comparison)

llama-batched-bench, 512 in / 128 out per stream, vs bartowski/nex-agi_Nex-N2-mini-GGUF Q4_K_M (19.91 GiB):

Prompt processing is ~10% faster on NVFP4 at every batch size (~9900–10200 vs ~8600–9300 tok/s). Single-stream decode still slightly favors stock Q4_K_M's MMQ kernel; from 4 concurrent streams up — the realistic serving regime — the NVFP4 variant wins on total throughput while using 1.5 GiB less VRAM.

(This is the first MoE release where our NVFP4 expert path beats stock K-quants in batched serving — earlier this year the MMQ kernel still won; upstream NVFP4 MoE optimization has since closed the gap.)

Long context on a single GPU

The hybrid linear-attention architecture keeps the KV cache small (only 10 of 40 layers carry full-attention KV, 2 KV heads), so the full 256k context plus vision fits a single RTX 5090 with room to spare:

Usage

Text-only (CLI)

llama-cli -m Nex-N2-mini-NVFP4-Q4_K_M.gguf -ngl 999 -c 8192 -p "Your prompt here"

Multimodal (server, vision + text)

llama-server \
  -m Nex-N2-mini-NVFP4-Q4_K_M.gguf \
  --mmproj mmproj-Nex-N2-mini-F16.gguf \
  -ngl 999 -c 32768 \
  --host 0.0.0.0 --port 8080

Then POST to /v1/chat/completions with image content blocks — see the llama.cpp multimodal docs.

Thinking model — fixed chat template included

Nex-N2 uses "Agentic Thinking" with adaptive reasoning depth — the chat template enables <think> blocks by default.

These GGUFs embed a fixed chat template. The upstream nex-agi template prefills the assistant turn with '<think>' (no trailing newline) while rendering past assistant reasoning as '<think>\n…'. That inconsistency breaks llama.cpp's reasoning extraction: the parser never recognizes the forced-open think block, so the full chain-of-thought (and a stray </think>) leaks into content instead of reasoning_content — on every llama.cpp build, regardless of --reasoning-format. Other community GGUFs of this model embed the upstream template and inherit the bug. Our embedded template adds the missing newline, so reasoning_content / content separation and tool-call parsing work out of the box with stock llama-server --jinja.

About the architecture

Nex-N2-mini is built on the Qwen3.5-MoE architecture (qwen35moe in GGUF): a hybrid linear-attention MoE with 40 layers (3 of every 4 layers use linear attention, every 4th is full attention), 256 experts (8 active per token) plus a shared expert, totalling 35B parameters with ~3B active. The upstream config declares a 1-layer MTP head, but the published checkpoints do not include MTP weights, so no MTP/speculative variant can be produced from public weights. The ModelOpt source keeps attention projections, linear-attention blocks, embeddings, and lm_head at BF16 — routed + shared expert FFNs (40 layers × 6 tensors) are NVFP4. The variants above differ only in how those non-expert tensors are stored.

Sources & credits

• Base model: nex-agi/Nex-N2-mini by Nex AGI — Apache 2.0
• NVFP4 calibration source: r0b0tlab/nex-n2-mini-nvfp4 (NVIDIA ModelOpt v0.44)
• mmproj source: official BF16 weights from nex-agi/Nex-N2-mini
• Stock baseline for benchmarks: bartowski/nex-agi_Nex-N2-mini-GGUF
• Regular (non-NVFP4) imatrix quants with the same template fix: LibertAIDAI/Nex-N2-mini-GGUF
• Tooling: llama.cpp convert_hf_to_gguf.py and llama-quantize

License

Apache 2.0, inherited from the upstream model.