Nex-N2.5-Pro-GGUF

Nex-N2.5-Pro GGUF

GGUF quantizations of nex-agi/Nex-N2.5-Pro, made with llama.cpp.

Chinese version: README_zh.md

397B total parameters, 60 layers in a hybrid layout: 45 Gated DeltaNet linear-attention layers and 15 full-attention layers (every fourth layer; GQA with 32 query / 2 KV heads, head dim 256). Each token is routed to 10 of 512 experts plus a shared expert. Partial RoPE (25 % of the head dim) with interleaved multimodal RoPE, 262,144-token context, 248,320-token vocabulary. A 27-layer vision encoder (768 px input, 16 px patches) ships here as a separate mmproj file.

Quantized from the official FP8 release, in which the routed experts are FP8 (E4M3, 128×128 blocks) and everything else is BF16. The FP8 experts are first dequantized exactly into a BF16 master; every tier is then quantized from that master with an importance matrix, and ships as 19 shards.

Quantizations

The master row is not a file in this repo. It is listed so the numbers above have a reference point — the BF16 master was loaded and measured on the same machine, so every tier is compared against a real baseline rather than an estimate.

Pick IQ4_XS unless you cannot fit it. On identical text its perplexity is only 0.8 % above the BF16 master's, it keeps the master's top-1 token on 94 % of positions, and its KL divergence is 4.3× lower than the next tier down. Q2_K is the balanced choice and the best option for CPU-only inference. IQ2_XS exists for when the footprint is hard-capped — even there, the median token's KL divergence stays at 0.05.

What is protected

The routed experts are 97.5 % of this model's weights, so everything else can be kept at high precision cheaply — measured with --dry-run, it costs +3.0 / +4.1 / +4.8 GiB over the bare IQ4_XS / Q2_K / IQ2_XS tiers:

One MTP (multi-token prediction) layer exists in the checkpoint but is excluded at conversion time via --no-mtp; these files carry the main model's tensors only.

Vision

The vision projector ships in BF16 and was checked end to end on the IQ4_XS weights with a synthetic test image (a red circle, a blue square and the text "QTUM 2026"): every shape, color and position was identified correctly, and the text was transcribed exactly.

Pass --image-min-tokens 1024. At llama.cpp's default budget a 768 px image is encoded into only 576 image tokens, which in our test was not enough to resolve fine print — the model read "QTUM" as "OTUM" — while with --image-min-tokens 1024 it read it correctly. llama.cpp itself warns that Qwen-VL-family models need at least 1024 image tokens. llama-mtmd-cli also needs --jinja to use the model's own chat template; it is off by default there.

Usage

# Text. Point at the first shard; llama.cpp finds the rest on its own.
# Sampling recommended by nex-agi: temperature 0.7, top-p 0.95, top-k 40.
llama-cli -m Nex-N2.5-Pro-IQ4_XS-00001-of-00019.gguf --temp 0.7 --top-p 0.95 --top-k 40

# Thinking is adaptive by default. Turn it off, or force it, through the chat template:
llama-cli -m Nex-N2.5-Pro-IQ4_XS-00001-of-00019.gguf --reasoning-effort none   # or: high

# Images: add the vision projector. --jinja is required here (off by default in llama-mtmd-cli).
llama-mtmd-cli -m Nex-N2.5-Pro-IQ4_XS-00001-of-00019.gguf \
  --mmproj mmproj-Nex-N2.5-Pro-BF16.gguf --jinja --image-min-tokens 1024 \
  --image photo.jpg -p "Describe this image."

Do not pass -ngl manually. llama.cpp fits layers to free VRAM by itself, and an explicit -ngl — including 0 — aborts that fitting and can fail to allocate compute buffers even when the weights would have fit.

Every tier ships as 19 shards. Download all 19 into one directory — you only ever name -00001-of-00019 on the command line. Requires a llama.cpp build with Qwen3.5-MoE (qwen35moe) support.

About the numbers

wikitext-2 test, n_ctx=512. Every figure in the table — the BF16 master's PPL included — is computed on the same 200 chunks of 512 tokens through the exact same command on 8× H100 80GB, so every row is compared on identical text. KL divergence and top-1 agreement are measured against logits saved from the BF16 master over those same chunks.

These numbers are only comparable within this table. Do not compare them against PPL figures published by other repos — different corpora and chunk counts make absolute values meaningless across setups.

Perplexity is a mean statistic that hides where a quantized model actually breaks, so KL divergence and top-1 agreement are reported alongside it.

License

Apache-2.0, inherited from nex-agi/Nex-N2.5-Pro — see the Apache License 2.0 for terms. Quantized by qtum.