MiMo-V2.6-Flash-RL-NVFP4

<p align="center"> <img src="https://img.shields.io/badge/size-182.40%20GB-F98424?style=flat-square" alt="size 182.40 GB"> <img src="https://img.shields.io/badge/experts-NVFP4%20W4A4-F98424?style=flat-square" alt="experts NVFP4 W4A4"> <img src="https://img.shields.io/badge/tool--calling-level%20with%20the%20release-3DD13D?style=flat-square" alt="tool-calling level with the release"> <img src="https://img.shields.io/badge/throughput%[email protected]×%20the%20release-3DD13D?style=flat-square" alt="1.32x the release throughput at concurrency 16"> <img src="https://img.shields.io/badge/GPUs-2×%2096%20GB-F98424?style=flat-square" alt="two 96 GB GPUs"> <img src="https://img.shields.io/badge/requires-Blackwell-lightgrey?style=flat-square" alt="requires Blackwell"> <a href="https://primitive.com"><img src="https://img.shields.io/badge/web-primitive.com-lightgrey?style=flat-square" alt="primitive.com"></a> </p> <h3 align="center">Xiaomi's own 4-bit experts, re-encoded for Blackwell's FP4 tensor cores.</h3> <p align="center"> NVFP4 build of <a href="https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL">XiaomiMiMo/MiMo-V2.6-Flash-RL</a> at <b>182.40 GB</b>, on two 96 GB cards.<br> Every routed-expert weight keeps its released value; qkv stays FP8 and everything else BF16, as released. </p>

There is no BF16 to start from. Xiaomi ships this model's routed experts as MXFP4 and its attention as FP8, so the release is already a 4-bit checkpoint and it is the reference row in every table below. This build changes the expert format, not the expert values.

---

Why this quant

• 🔁 The release's weights, bit for bit. All 302,795,194,368 routed-expert values reproduce the released MXFP4 values exactly. Each E8M0 scale over 32 weights becomes two E4M3 scales over 16 against a power-of-two global, and no value needed rounding.
• ⚡ The experts run on native FP4. vLLM serves them with FlashInfer CUTLASS W4A4 on sm_120. Over the full knowledge suite at concurrency 16 that is 532 tok/s against the release's 402.
• 🎯 Tool-calling level with the release. 78.6 over five runs, the release also 78.6. Knowledge 91.1 against 90.3.
• 📏 Activation scales set per layer. FP4 activations need one global scale per layer. The last MoE layer's expert intermediate reaches about 60,000 on the prompts we measured, 22× past what a scale of 1.0 can hold, so a single constant would clip it.
• 🧮 The cost is KV cache. NVFP4 stores 4.5 bits per weight to MXFP4's 4.25, so this build is 9.48 GB larger than the release. On two 96 GB cards that leaves a 250,094-token pool, against 627,110 for the release.

---

Serve it

hf download primitive-ai/MiMo-V2.6-Flash-RL-NVFP4 --local-dir ./MiMo-V2.6-Flash-RL-NVFP4

SP=/usr/local/lib/python3.12/dist-packages/vllm
docker run --gpus all --ipc=host --shm-size 32g -p 8000:8000 -v $PWD:/models \
  -v $PWD/MiMo-V2.6-Flash-RL-NVFP4/vllm_patch/mimo_v2.py:$SP/model_executor/models/mimo_v2.py:ro \
  vllm/vllm-openai:mimo-v26-x86_64-cu130 \
  --model /models/MiMo-V2.6-Flash-RL-NVFP4 \
  --tensor-parallel-size 2 --trust-remote-code --generation-config vllm \
  --max-model-len 131072 --gpu-memory-utilization 0.95 \
  --limit-mm-per-prompt '{"image":4,"video":0,"audio":0}' \
  --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice

The day-0 image (vLLM 0.29.1rc1.dev449) loads ModelOpt mixed-precision checkpoints, but its MiMo loader assumes the release's own FP8 layout: the fused qkv loader writes to weight_scale_inv, while ModelOpt's block-FP8 layers name that parameter weight_scale and store it 4-D. vllm_patch/mimo_v2.py is the image's file with those spots bridged; the 21-line diff is vllm_patch/mimo_v2.diff. Without the mount, loading stops with a KeyError.

Setting audio to 0 skips loading the audio encoder. The DFlash drafter ships as released, but speculative decoding was not measured with this build.

---

Measured

Two RTX PRO 6000 Blackwell, 96 GB each, tensor parallel 2. The 1,170-item knowledge suite and the 200-item tool-calling suite, temperature 0.6 / top_p 0.95 / top_k 20, thinking on, a 16,384-token budget, concurrency 16, auto-scored with no LLM judge. bf16 KV, no speculative decoding, same box, same day.

Tool-calling is the mean of five runs per build, with a within-build standard deviation of 0.8 to 0.9, so the column is one band. Knowledge is one run per quantized build and two for the release; gaps under 1.2 are ties. tok/s @1 is 60 real prompts one at a time; tok/s @16 is the knowledge suite's own aggregate.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

MiMo-V2.6-Flash-RL is 309 B parameters, 15 B active, and 98% of those parameters are routed experts.

ModelOpt MIXED_PRECISION, one quantized_layers entry per module; the dense FP8 entries are tagged FP8_PB_WO, the name vLLM's ModelOpt linear table accepts for block FP8. gate and up of each expert share one global scale, so the fused projection vLLM builds keeps the exact values. vLLM reduces expert input scales to one per layer, and the checkpoint stores them that way.

The expert weights are not re-quantized. The activation scales are the only new numbers in this checkpoint.

---

<p align="center"> <br> <img src="https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8/resolve/main/assets/primitive-logo.png" alt="Primitive" width="34"><br> _{<a href="https://primitive.com"><b>primitive</b></a> · <a href="https://huggingface.co/primitive-ai">more models</a> · inference economics for production LLM systems} </p>