MiMo-V2.5-ASR-FP8

MiMo-V2.5-ASR — FP8 (e4m3fn)

FP8-quantized build of XiaomiMiMo/MiMo-V2.5-ASR, the Xiaomi MiMo end-to-end ASR model with native Mandarin/English code-switching, Chinese dialects, song lyrics, noisy/multi-speaker robustness, and native punctuation.

What this is

• Weights: float8_e4m3fn, per-output-channel absmax scaling (one fp32 scale per row), baked at save time.
• Activations: dynamic per-tensor fp8 quantization each forward pass.
• Matmul: torch._scaled_mm (FP8 tensor cores on Ada / Hopper / Blackwell).
• Skipped (kept bf16): embeddings, RMSNorm/LayerNorm, biases. 417 Linear layers converted.
• The audio encoder/tokenizer (MiMo-Audio-Tokenizer) is not quantized; download it separately for inference.

This roughly halves the LLM weight footprint (~32 GB bf16 → ~16-17 GB on disk).

Important: this is NOT a drop-in from_pretrained checkpoint

model.safetensors stores custom FP8Linear buffers (*.weight_fp8, *.weight_scale), not standard HF Linear weights. It must be loaded through the matching FP8Linear modules. Use the loader below.

Usage

git clone https://github.com/XiaomiMiMo/MiMo-V2.5-ASR.git
cd MiMo-V2.5-ASR
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1   # required by the audio tokenizer
hf download XiaomiMiMo/MiMo-Audio-Tokenizer --local-dir ./models/MiMo-Audio-Tokenizer
hf download Infatoshi/MiMo-V2.5-ASR-FP8 --local-dir ./MiMo-V2.5-ASR-FP8

Then install the maintained loader from RESMP-DEV/mimo-asr-fp8 and load with its FP8Linear implementation:

from quantize_fp8 import load_fp8_model
mimo = load_fp8_model(
    fp8_dir="./MiMo-V2.5-ASR-FP8",
    tokenizer_path="./models/MiMo-Audio-Tokenizer",
    repo_root=".",                     # the cloned MiMo-V2.5-ASR repo
)
print(mimo.asr_sft("audio.wav", audio_tag="<english>"))

Quantization fidelity

Per-output-channel absmax dequant error vs the original fp32 weights, sampled across depth (layers 0/17/35), all attn+mlp projections, lm_head, and the audio local transformer:

• relative Frobenius error: ~0.026, uniform across every sampled layer (max 0.027 on lm_head)
• no corrupted or outlier layers

This is the expected magnitude for fp8 e4m3 with per-channel scaling (3 mantissa bits).

Requirements

• CUDA GPU with FP8 tensor cores (Ada / Hopper / Blackwell), CUDA >= 12.0
• torch >= 2.6, safetensors
• Blackwell (sm_120, e.g. RTX PRO 6000 / RTX 50xx): use a torch build with CUDA 12.8+ (torch >= 2.7, cu128). torch 2.6 cu124 ships no sm_120 kernels and will fail with "no kernel image is available for execution on the device".

Notes / caveats

• FP8 e4m3fn weight-only-style quantization is lossy; expect small WER deltas vs bf16.
• Per-tensor dynamic activation scaling is simple and fast but less accurate than per-token scaling on activations with large outliers.

Derivative of an MIT-licensed model; original credit to the Xiaomi MiMo team.