MiniCPM-o-4_5-BNB-Int4

MiniCPM-o 4.5 — INT4 (bitsandbytes NF4)

4-bit bitsandbytes quantization of openbmb/MiniCPM-o-4_5 with selective module skipping for audio/vision quality preservation.

Highlights

• Full multimodal capability preserved: text, vision, audio input, TTS output, voice cloning
• Only LLM transformer layers are quantized — audio encoder (Whisper), vision encoder (SigLIP), TTS decoder, and projection layers remain in bf16
• TTS weight_norm layers are explicitly skipped (they crash under bitsandbytes quantization)
• Tested with 293 unit + integration tests — all passing
• Benchmark-validated: text quality identical to bf16, audio quality within natural variation

VRAM Requirements

Benchmarked on NVIDIA RTX PRO 6000 Blackwell (96 GB). Your load times will vary.

Quick Start

from transformers import AutoModel, AutoTokenizer, BitsAndBytesConfig
import torch

model_name = "ericleigh007/MiniCPM-o-4_5-BNB-Int4"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    init_vision=True,
    init_audio=True,
    init_tts=True,
)
model.eval()
model.init_tts()

Quantization Details

Method: bitsandbytes NF4 with double quantization

Modules kept in bf16 (not quantized):

Benchmark Results

Text quality is identical to bf16 (echo prompts return exact matches, math returns correct answers). Audio quality shows minor spectral variation within the range of normal run-to-run differences.

Full benchmark report with spectrograms: OmniChat Quantization Benchmarks

License

Same as the base model: MiniCPM Model License

Credits

• Base model by OpenBMB
• Quantization and testing by ericleigh007
• Part of the OmniChat project

---

Exported on 2026-03-11