Nex-N2.5-mini-mixed-NVFP4-FP8

<p align="center"> <img src="https://img.shields.io/badge/size-26.04%20GB-F98424?style=flat-square" alt="size 26.04 GB"> <img src="https://img.shields.io/badge/experts-34%20NVFP4%20%2B%206%20FP8%20layers-F98424?style=flat-square" alt="34 NVFP4 layers and 6 FP8 layers"> <img src="https://img.shields.io/badge/accuracy-level%20with%20BF16-3DD13D?style=flat-square" alt="accuracy level with BF16"> <img src="https://img.shields.io/badge/drift%20vs%20BF16-20%25%20lower%20than%20plain%20NVFP4-3DD13D?style=flat-square" alt="20 percent lower representation drift than plain NVFP4"> <img src="https://img.shields.io/badge/KV%20pool-2.65M%20tokens-F98424?style=flat-square" alt="KV pool 2.65M tokens"> <img src="https://img.shields.io/badge/requires-Blackwell-lightgrey?style=flat-square" alt="requires Blackwell"> <a href="https://primitive.com"><img src="https://img.shields.io/badge/web-primitive.com-lightgrey?style=flat-square" alt="primitive.com"></a> </p> <h3 align="center">Six of forty expert layers at 8 bits.</h3> <p align="center"> Mixed-precision quantization of <a href="https://huggingface.co/nex-agi/Nex-N2.5-mini">nex-agi/Nex-N2.5-mini</a> to <b>26.04 GB</b>, on one 96 GB card.<br> 34 of the 40 routed-expert layers at NVFP4 and six at FP8; attention, linear attention, shared experts, embeddings and the vision tower stay BF16. </p>

Three builds, one measurement. We quantized this model's experts three ways and measured all of them against the BF16 original under one protocol: this one, an NVFP4 build at 23.91 GB, an FP8 build at 38.13 GB, and this one. On accuracy the four are one band, and this card says so rather than claiming a win the measurement does not support. Where this build does differ is in how far its representations sit from the BF16 original, and in landing between the two endpoints on both size and speed.

---

Why this quant

• 📉 Closest to the original of the three quantized builds. Its six FP8 layers remove 19.7% of the representation drift that 4-bit quantization introduces, for six layers of forty.
• 🎯 Accuracy is level with BF16, like every build here. 90.6 knowledge and 76.5 tool-calling against the original's 89.6 and 75.3.
• ⚖️ Between the endpoints on both axes. 26.04 GB against 23.91 and 38.13; 147 tok/s single-stream and 990 at concurrency 32, in both cases between the 4-bit and 8-bit builds.
• 🧩 Stock vLLM, no patches. vLLM resolves the compressed-tensors scheme per MoE layer and runs both kernels in one model: FlashInfer CUTLASS for the NVFP4 layers, Triton FP8 for the other six.

---

Serve it

hf download primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8 --local-dir ./Nex-N2.5-mini-mixed-NVFP4-FP8

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  vllm/vllm-openai:v0.28.0 \
  --model /models/Nex-N2.5-mini-mixed-NVFP4-FP8 \
  --max-model-len 32768 --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Qwen3_5MoeForConditionalGeneration is in the 0.28.0 release, so no nightly is needed. The numbers on this page were measured on 0.28.1rc1.dev472. We booted this build on 0.28.0 as well: 181 s to healthy, both MoE kernels selected as expected, generation sane.

Two things about this model's chat template are worth knowing before you wire it up. It emits an empty thinking block by default, so replies arrive with two leading newlines and the reasoning field holds only whitespace; a short correct answer looks like "\n\n4". And reasoning_effort (low / medium / xhigh) is a per-request field here, unlike some other reasoning models where it is fixed at server start.

---

Measured

One RTX PRO 6000 Blackwell, 96 GB, one card. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache free, two seeds per cell. Every row below was run on the same box in the same sitting.

Tool-calling is the mean of two runs per build (three for BF16). Pooled within-build spread is 0.60 standard deviations over those nine runs, and the four build means span 1.7, so the column is one band. Read the ordering with that in mind: it runs opposite to precision, which no mechanism would predict, and is the clearest evidence that this suite cannot resolve the difference between 4-bit and 16-bit experts on this model.

Speed does separate, and it crosses over. FP8 experts decode 13% faster single-stream because the 4-bit path pays for activation quantization and group scales on every token, a cost that does not amortize at batch 1. At concurrency 32 the ranking inverts and NVFP4 wins by 13%, because there the bottleneck is weight bandwidth and 4-bit moves half the bytes. Pick by the concurrency you actually run.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

Nex-N2.5-mini is 35.11 B parameters, and they are distributed very unevenly:

compressed-tensors, format mixed-precision, two config groups whose targets name specific layers. vLLM resolves the scheme per MoE layer, so interleaving is fine, but all three projections of one layer must agree, which is why the unit of allocation is a whole layer. The source ships experts as two batched 3-D tensors per layer; they are unfolded into per-expert modules, which is the layout vLLM's loader consumes. gate and up share one weight_global_scale per expert. vLLM fuses those two halves and keeps a single scale, warning and taking the maximum when they disagree, which silently shrinks half the weights, so the build is audited for that before release: 20,480 fused groups checked, no mismatches.

Weights-only round-to-nearest, no calibration. Nothing in the quantization is fitted to data.

---

<p align="center"> <br> <img src="https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8/resolve/main/assets/primitive-logo.png" alt="Primitive" width="34"><br> _{<a href="https://primitive.com"><b>primitive</b></a> · <a href="https://huggingface.co/primitive-ai">more models</a> · inference economics for production LLM systems} </p>