MiMo-V2.5-NVFP4-Experts

MiMo-V2.5-NVFP4-Experts

Expert-only NVFP4 quantization of XiaomiMiMo/MiMo-V2.5 (310B MoE, 15B active params).

This is the non-Pro variant — MiMo-V2.5 (310B, hidden_size 4096), quantized identically to how Xiaomi produced their MiMo-V2.5-Pro-FP4-DFlash.

What Was Done

Quantization Approach

MiMo-V2.5 is a Mixture-of-Experts (MoE) model with 256 routed experts per layer. Only the MoE expert weights are quantized to NVFP4 (4-bit NormalFloat), while attention projections, vision encoder (729M), audio encoder (261M), and other modules remain at higher precision.

This matches Xiaomi's own approach for the Pro version:

"We quantize only the MoE experts to FP4 (MXFP4) and keep the other modules at their original precision." — MiMo-V2.5-Pro-FP4-DFlash

Model Summary

Quantization Config

{
  "quantization_method": "nvfp4_experts",
  "kv_cache_dtype": "fp8",
  "quantization_library": "nvidia-modelopt",
  "calibration_samples": 64
}

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "gaber/MiMo-V2.5-NVFP4-Experts",
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(
    "gaber/MiMo-V2.5-NVFP4-Experts",
    trust_remote_code=True,
)

Deployment

SGLang

python3 -m sglang.launch_server \
    --model-path gaber/MiMo-V2.5-NVFP4-Experts \
    --quantization fp8 \
    --trust-remote-code \
    --dtype bfloat16 \
    --context-length 32768

vLLM

vllm serve gaber/MiMo-V2.5-NVFP4-Experts \
    --quantization fp8 \
    --trust-remote-code \
    --dtype bfloat16 \
    --max-model-len 32768

Hardware Requirements

• Minimum VRAM: ~125 GB (for quantized weights alone)
• Recommended: 2× H100/H200 or 4× A100 80GB
• Single H200 NVL: Works with device_map="auto" (VRAM + RAM split)

Limitations

• DFlash Drafter: Not included. Xiaomi's DFlash draft model targets MiMo-V2.5-Pro (hidden_size 6144), which is incompatible with this base model (hidden_size 4096).
• MTP Heads: Untouched (3-layer multi-token prediction preserved)
• Vision/Audio: Untouched (729M ViT + 261M Audio encoders preserved)

Base Model

XiaomiMiMo/MiMo-V2.5 — 310B MoE, 15B active, hybrid SWA/GA attention, 1M context, multimodal (text/image/video/audio).

Quantization Tool

NVIDIA ModelOpt 0.44.0 — NVFP4 experts-only quantization with 64-sample calibration.

License

Same as base model: MIT