MiMo-V2.5-0703-NVFP4

MiMo-V2.5-0703-NVFP4

NVFP4/MXFP8 mixed quantization of the updated MiMo-V2.5 target model that ships inside XiaomiMiMo/MiMo-V2.5-DFlash (released 2026-07-03) — the exact weights the DFlash speculative-decoding drafter was trained against.

Why this exists: the widely used lukealonso/MiMo-V2.5-NVFP4 was quantized from the original April XiaomiMiMo/MiMo-V2.5 release. The 2026-07-03 DFlash drop silently includes a refreshed target: every weight shard differs (MoE router gates, routed experts, attention o_proj; the vision/audio towers and audio tokenizer are unchanged). Serving the DFlash drafter against the old base works, but leaves drafter/target mismatch on the table. This repo is the drafter-matched target.

What it is

• Same recipe, layout and tensor structure as lukealonso/MiMo-V2.5-NVFP4 (verified structurally identical tensor-by-tensor: names, dtypes, shapes) — a drop-in replacement wherever that checkpoint serves.
• Routed MoE experts: NVFP4 (E2M1, group-16 FP8 scales + per-tensor scale, calibrated input scales). Attention fused-QKV and dense MLP: MXFP8 (group-32 E8M0 scales, exact conversion from the upstream FP8 release). o_proj, routers, embeddings, lm_head, norms, vision/audio towers: BF16. MTP draft weights carried (model-mtp.safetensors).
• Calibration: seeded with lukealonso's published full 6-pass amax state (text/image/audio/video coverage), then adapted to the new weights with fresh passes (agentic coding, diverse chat, VQA, long-context). Weight scales computed entirely from the new weights.
• Quantized with the open pipeline in local-inference-lab/quant-toolkit.

Running with vLLM (SM120 / RTX PRO 6000 Blackwell)

Use the companion image (custom kernels for this format + DFlash):

docker pull ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash

The DFlash drafter is the dflash/ subfolder of the upstream repo — download it and pass the subfolder as the speculative model:

hf download XiaomiMiMo/MiMo-V2.5-DFlash --include "dflash/*" --local-dir /models/mimo-dflash

docker run --gpus '"device=0,1,2,3"' --ipc host --network host --init --shm-size 32g \
  -v $HF_CACHE:/root/.cache/huggingface \
  -v /models/mimo-dflash/dflash:/dflash:ro \
  -v mimo-jit-cache:/cache \
  -e VLLM_USE_V2_MODEL_RUNNER=0 \
  -e VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SAFETENSORS_FAST_GPU=1 \
  -e CUDA_DEVICE_MAX_CONNECTIONS=32 \
  -e OMP_NUM_THREADS=16 \
  -e CUTE_DSL_ARCH=sm_120a \
  -e NCCL_P2P_DISABLE=0 \
  -e NCCL_SHM_DISABLE=0 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_MIN_NCHANNELS=8 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_CUMEM_HOST_ENABLE=0 \
  ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash \
  vllm serve mitomtuna/MiMo-V2.5-0703-NVFP4 \
    --served-model-name mimo-v2.5 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --kv-cache-dtype auto \
    --block-size 64 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 8192 \
    --max-cudagraph-capture-size 32 \
    --attention-backend TRITON_ATTN \
    --kernel-config.moe_backend flashinfer_cutlass \
    --kernel-config.linear_backend b12x \
    --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice \
    --compilation-config '{"cudagraph_mode":"PIECEWISE","custom_ops":["all"]}' \
    --async-scheduling --no-scheduler-reserve-full-isl \
    --enable-chunked-prefill --enable-prefix-caching \
    --disable-custom-all-reduce \
    --speculative-config '{"model":"/dflash","method":"dflash","num_speculative_tokens":7,"num_speculative_tokens_per_batch_size":[[1,4,7],[5,32,3]]}'

This is the exact production configuration these numbers were measured on (same env, same flags; only host/port and model paths differ — we bind-mount the checkpoint instead of pulling from HF, which changes nothing).

Notes:

• Read "Host setup" below first — without driver-enabled GPU P2P the decode throughput in the validation section is not reproducible.
• Optional --kv-cache-dtype fp8 (needs a 2026-07-13+ image pull; older builds silently ignore the flag): ~2× KV pool, acceptance at parity, but cold long-context prefill drops ~30-50% at 128k+ (the b12x prefill fast path is 16-bit-only). Keep auto unless you need the bigger pool.
• The checkpoint is self-contained (auto_map + configuration_mimo_v2.py included): no config shim needed.
• Text, image, audio and video inputs all work; at TP4 no special --mm-encoder-tp-mode is needed.
• Need 3 GPUs? Use the exact tensor-parallel-3 transform of this repo: mitomtuna/MiMo-V2.5-0703-NVFP4-TP3.

Host setup (required for full multi-GPU speed)

Tensor-parallel decode is allreduce-latency-bound. On RTX PRO 6000 Blackwell the driver disables PCIe peer access by default, so NCCL silently falls back to shared-memory copies and decode is dramatically slower. To get the fast lane the validation numbers were measured on:

1. Force P2P in the driver — /etc/modprobe.d/nvidia-p2p-override.conf:

   options nvidia NVreg_RegistryDwords="ForceP2P=0x11;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"
   

   then update-initramfs -u and reboot. Do not add the older RMForceP2PType=1;RMPcieP2PType=2 dwords seen in some recipes — they break cross-process cuMem IPC on NCCL ≥ 2.29.
2. BIOS: Resizable BAR and Above-4G decoding ON; IOMMU OFF (or ACS override) — an active IOMMU degrades or breaks PCIe P2P.
3. Container env (in the command above): NCCL_P2P_LEVEL=SYS so NCCL uses P2P across PCIe root complexes, and NCCL_CUMEM_ENABLE=0 (legacy cudaIpc transport — most reliable with forced P2P).
4. Keep --disable-custom-all-reduce: plain NCCL over P2P is the fast path on this platform (vLLM's custom-allreduce kernels crash or regress here).

Verify: nvidia-smi topo -p2p r should show all GPU pairs OK, and a boot with NCCL_DEBUG=INFO should log channels via P2P, not via SHM.

Credits

• Original model + DFlash drafter + updated target: XiaomiMiMo
• Quantization recipe, calibration seed and pipeline: lukealonso / local-inference-lab

Validation (this exact checkpoint, RTX PRO 6000 Blackwell)

0703-target requant validation — 2026-07-08T07:40:08+00:00

Acceptance (reason_accept_check.py, same prompt/protocol)

--- baseline: prod TP3, OLD target (captured in prep):
reasoning ctx0: 412.3 t/s, accept_len 5.91/8 (2500 tokens, 423 drafts)
--- mimo-vllm-val (NEW target):
reasoning ctx0: 488.0 t/s, accept_len 5.69/8 (2500 tokens, 439 drafts)
--- mimo-tp3-val (NEW target):
reasoning ctx0: 381.5 t/s, accept_len 5.71/8 (2500 tokens, 438 drafts)

Needle (90k)

--- mimo-vllm-val:
prompt_tokens=89935 completion=99
answer: 'PERSIMMON-42'
NEEDLE PASS
--- mimo-tp3-val:
prompt_tokens=89935 completion=103
answer: 'PERSIMMON-42'
NEEDLE PASS

KV cache pool measured at boot (this checkpoint, this command)

--- TP4 (util 0.85, max-model-len 1048576):
(Worker_TP0 pid=247) INFO 07-08 07:35:33 [gpu_worker.py:517] Available KV cache memory: 29.77 GiB
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2202] GPU KV cache size: 3,877,291 tokens
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 3.70x
--- TP3 sibling, bf16 KV (util 0.965, max-model-len 524288):
(Worker_TP0 pid=241) INFO 07-08 07:39:30 [gpu_worker.py:517] Available KV cache memory: 14.9 GiB
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2202] GPU KV cache size: 668,489 tokens
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2203] Maximum concurrency for 524,288 tokens per request: 1.28x
--- TP3 sibling, fp8 KV (util 0.965, max-model-len 1048576 — needs 2026-07-13+ image):
(Worker_TP0 pid=186) INFO 07-13 06:46:02 [gpu_worker.py:517] Available KV cache memory: 14.87 GiB
(EngineCore pid=118) INFO 07-13 06:46:02 [kv_cache_utils.py:2202] GPU KV cache size: 1,358,830 tokens
(EngineCore pid=118) INFO 07-13 06:46:02 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 1.30x

Smoke responses: see s5-smoke-*.json

Result: ALL PASS