MiMo-V2.5-0703-NVFP4-TP3

MiMo-V2.5-0703-NVFP4-TP3

mitomtuna/MiMo-V2.5-0703-NVFP4 transformed to run at tensor-parallel size 3 (e.g. 3× 96 GB GPUs) — a geometry the source cannot serve (64 attention heads, 4/8 KV heads, MoE intermediate 2048: none divisible by 3).

This is the updated 2026-07-03 MiMo-V2.5 target (the exact weights the DFlash drafter was trained against) — the successor of mitomtuna/MiMo-V2.5-NVFP4-TP3, which was built from the original April base.

The transform is mathematically exact — no requantization, no fine-tuning. Full-attn 64Q/4KV → 72Q/9KV (bit-identical KV duplication, slot map [0,0,0,1,1,2,2,3,3]), SWA 64Q/8KV → 72Q/9KV ([0,0,1,2,3,4,5,6,7]), padded Q heads have zeroed q rows AND zeroed o_proj columns (exact-zero contribution), MoE expert intermediate 2048 → 2304 and dense MLP 16384 → 16512 via group-aligned zero padding of the packed NVFP4/MXFP8 tensors. All original quantization groups are byte-identical. transform_checkpoint.py (included) is the exact script that produced this repo.

Use with the matching DFlash drafter (same transform, drafter weights are Xiaomi's originals): mitomtuna/MiMo-V2.5-DFlash-TP3.

Running with vLLM

docker pull ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash

docker run --gpus '"device=0,1,2"' --ipc host --network host --init --shm-size 32g \
  -v $HF_CACHE:/root/.cache/huggingface \
  -v mimo-jit-cache:/cache \
  -e VLLM_USE_V2_MODEL_RUNNER=0 \
  -e VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e SAFETENSORS_FAST_GPU=1 \
  -e CUDA_DEVICE_MAX_CONNECTIONS=32 \
  -e OMP_NUM_THREADS=16 \
  -e CUTE_DSL_ARCH=sm_120a \
  -e NCCL_P2P_DISABLE=0 \
  -e NCCL_SHM_DISABLE=0 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_MIN_NCHANNELS=8 \
  -e NCCL_CUMEM_ENABLE=0 \
  -e NCCL_CUMEM_HOST_ENABLE=0 \
  ghcr.io/tunamitom/mimo-vllm:cu132-nvfp4-dflash \
  vllm serve mitomtuna/MiMo-V2.5-0703-NVFP4-TP3 \
    --served-model-name mimo-v2.5 \
    --trust-remote-code \
    --tensor-parallel-size 3 \
    --kv-cache-dtype fp8 \
    --block-size 64 \
    --gpu-memory-utilization 0.965 \
    --max-model-len 1048576 \
    --max-num-seqs 32 \
    --max-num-batched-tokens 4096 \
    --max-cudagraph-capture-size 32 \
    --attention-backend TRITON_ATTN \
    --kernel-config.moe_backend flashinfer_cutlass \
    --kernel-config.linear_backend b12x \
    --mm-encoder-tp-mode data \
    --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice \
    --compilation-config '{"cudagraph_mode":"PIECEWISE","custom_ops":["all"]}' \
    --async-scheduling --no-scheduler-reserve-full-isl \
    --enable-chunked-prefill --enable-prefix-caching \
    --disable-custom-all-reduce \
    --speculative-config '{"model":"mitomtuna/MiMo-V2.5-DFlash-TP3","method":"dflash","num_speculative_tokens":7,"num_speculative_tokens_per_batch_size":[[1,4,7],[5,32,3]]}'

This is the exact production configuration the numbers below were measured on (only host/port and model paths differ).

Notes:

• Read "Host setup" below first — without driver-enabled GPU P2P the decode numbers here are not reproducible.
• fp8 KV (the command above) needs a 2026-07-13+ image pull — older builds silently ignore --kv-cache-dtype and the 1M pool won't fit. It gives 1,358,484 tokens (1.30× at 1M, 2.0× bf16), acceptance at parity, decode 7–12% faster at 128k+ — but cold long-context prefill 30–50% slower (the b12x prefill fast path is 16-bit-only). bf16 alternative: --kv-cache-dtype auto --max-model-len 524288 → 668,489 tokens, fastest prefill.
• 0.975 util OOMs a ~4.8 GiB init transient — 0.965 assumes dedicated, otherwise-idle 96 GB GPUs; back off to 0.93–0.95 if anything else shares them.
• --mm-encoder-tp-mode data is required at TP3 (vision tower has 32 heads, not divisible by 3; encoders run replicated). Text/image/audio/video all work.
• Decode throughput ≈ 0.75–0.86× of the same engine at TP4; KV duplication costs ~3× per-token KV per rank vs TP4 (fp8 KV halves that back).

Host setup (required for full multi-GPU speed)

Tensor-parallel decode is allreduce-latency-bound. On RTX PRO 6000 Blackwell the driver disables PCIe peer access by default, so NCCL silently falls back to shared-memory copies and decode is dramatically slower. To get the fast lane these numbers were measured on:

1. Force P2P in the driver — /etc/modprobe.d/nvidia-p2p-override.conf:

   options nvidia NVreg_RegistryDwords="ForceP2P=0x11;GrdmaPciTopoCheckOverride=1;EnableResizableBar=1"
   

   then update-initramfs -u and reboot. Do not add the older RMForceP2PType=1;RMPcieP2PType=2 dwords seen in some recipes — they break cross-process cuMem IPC on NCCL ≥ 2.29.
2. BIOS: Resizable BAR and Above-4G decoding ON; IOMMU OFF (or ACS override) — an active IOMMU degrades or breaks PCIe P2P.
3. Container env (in the command above): NCCL_P2P_LEVEL=SYS so NCCL uses P2P across PCIe root complexes, and NCCL_CUMEM_ENABLE=0 (legacy cudaIpc transport — most reliable with forced P2P).
4. Keep --disable-custom-all-reduce: plain NCCL over P2P is the fast path on this platform (vLLM's custom-allreduce kernels crash or regress here).

Verify: nvidia-smi topo -p2p r should show all GPU pairs OK, and a boot with NCCL_DEBUG=INFO should log channels via P2P, not via SHM.

Credits

• Original model + DFlash drafter + updated target: XiaomiMiMo
• Quantization recipe/seed/pipeline: lukealonso / local-inference-lab
• TP4→TP3 transform: transform_checkpoint.py in this repo

Validation (this exact checkpoint, RTX PRO 6000 Blackwell)

0703-target requant validation — 2026-07-08T07:40:08+00:00

Acceptance (reason_accept_check.py, same prompt/protocol)

--- baseline: prod TP3, OLD target (captured in prep):
reasoning ctx0: 412.3 t/s, accept_len 5.91/8 (2500 tokens, 423 drafts)
--- mimo-vllm-val (NEW target):
reasoning ctx0: 488.0 t/s, accept_len 5.69/8 (2500 tokens, 439 drafts)
--- mimo-tp3-val (NEW target):
reasoning ctx0: 381.5 t/s, accept_len 5.71/8 (2500 tokens, 438 drafts)

Needle (90k)

--- mimo-vllm-val:
prompt_tokens=89935 completion=99
answer: 'PERSIMMON-42'
NEEDLE PASS
--- mimo-tp3-val:
prompt_tokens=89935 completion=103
answer: 'PERSIMMON-42'
NEEDLE PASS

KV cache pool measured at boot (this checkpoint, this command)

--- TP4 (util 0.85, max-model-len 1048576):
(Worker_TP0 pid=247) INFO 07-08 07:35:33 [gpu_worker.py:517] Available KV cache memory: 29.77 GiB
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2202] GPU KV cache size: 3,877,291 tokens
(EngineCore pid=179) INFO 07-08 07:35:33 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 3.70x
--- TP3, fp8 KV (util 0.965, max-model-len 1048576 — current command above):
(Worker_TP0 pid=186) INFO 07-13 06:21:12 [gpu_worker.py:517] Available KV cache memory: 14.87 GiB
(EngineCore pid=118) INFO 07-13 06:21:12 [kv_cache_utils.py:2202] GPU KV cache size: 1,358,484 tokens
(EngineCore pid=118) INFO 07-13 06:21:12 [kv_cache_utils.py:2203] Maximum concurrency for 1,048,576 tokens per request: 1.30x
--- TP3, bf16 KV (util 0.965, max-model-len 524288 — the "auto" alternative):
(Worker_TP0 pid=241) INFO 07-08 07:39:30 [gpu_worker.py:517] Available KV cache memory: 14.9 GiB
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2202] GPU KV cache size: 668,489 tokens
(EngineCore pid=172) INFO 07-08 07:39:30 [kv_cache_utils.py:2203] Maximum concurrency for 524,288 tokens per request: 1.28x

Smoke responses: see s5-smoke-*.json

Result: ALL PASS