mplug2-vp-for-nriqa

Visual Prompt Checkpoints for NR-IQA

🔬 Paper: Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA 💻 Code: GitHub Repository

Abstract

In this paper, we propose a novel approach to No-Reference Image Quality Assessment (NR-IQA) by efficiently adapting a Multimodal Large Language Model (MLLM) through pixel-space visual prompts. Unlike full fine-tuning approaches that adapt MLLMs to specific tasks, our method trains only ∼600K parameters at most (<0.01% of the base model), while keeping the underlying model fully frozen. During inference, these visual prompts are combined with images via addition and processed by mPLUG-Owl2 with the textual query “Rate the technical quality of the image.” Evaluations across distortion types (synthetic, realistic, AI-generated) on KADID-10k, KonIQ-10k, and AGIQA-3k demonstrate competitive performance against full finetuned methods and specialized NR-IQA models, achieving 0.93 SRCC on KADID-10k. The source code is publicly available at https: // github. com/ yahya-ben/ mplug2-vp-for-nriqa .

Overview

Pre-trained visual prompt checkpoints for No-Reference Image Quality Assessment (NR-IQA) using mPLUG-Owl2-7B. Achieves competitive performance with only ~600K parameters vs 7B+ for full fine-tuning.

🔥 Key Features

• Parameter-Efficient: Only ~600K trainable parameters vs 7B+ for full fine-tuning
• Competitive Performance: Achieves 0.93 SROCC on KADID-10k dataset
• Multiple Visual Prompt Types: Padding, Fixed Patches (Center/Top-Left), Full Overlay
• Multiple MLLM Support: mPLUG-Owl2-7B
• Comprehensive Evaluation: Supports KADID-10k, KonIQ-10k, and AGIQA-3k datasets
• Pre-trained Checkpoints: Available on HuggingFace Hub for immediate use

Available Checkpoints

Download: visual_prompt_ckpt_trained_on_mplug2.zip

🏃 Usage

This section provides instructions for setting up the environment, preparing datasets, and running inference with the pre-trained visual prompt checkpoints. For detailed setup, training, and further usage instructions, please refer to the GitHub repository.

Prerequisites

• Python 3.10+
• CUDA-capable GPU (tested on NVIDIA RTX A6000)
• PyTorch
• HuggingFace Transformers

Setup Environment

For mPLUG-Owl2:

# Clone and setup mPLUG-Owl2
git clone https://github.com/X-PLUG/mPLUG-Owl.git
cd mPLUG-Owl/mPLUG-Owl2
conda create -n mplug_owl2 python=3.10 -y
conda activate mplug_owl2
pip install --upgrade pip
pip install -e .
pip install 'numpy<2'
pip install protobuf

Additional Dependencies:

pip install PyYAML scikit-learn tqdm

Dataset Setup

Download the required IQA datasets:

# KonIQ-10k
wget https://huggingface.co/datasets/chaofengc/IQA-PyTorch-Datasets/resolve/main/koniq10k.tgz
tar -xzf koniq10k.tgz

# KADID-10k  
wget https://huggingface.co/datasets/chaofengc/IQA-PyTorch-Datasets/resolve/main/kadid10k.tgz
tar -xzf kadid10k.tgz

# AGIQA-3K
wget https://huggingface.co/datasets/chaofengc/IQA-PyTorch-Datasets/resolve/main/AGIQA-3K.zip
unzip AGIQA-3K.zip

After extraction, organize your datasets in the data/ folder as follows:

data/
├── kadid10k/
│   ├── images/           # All KADID-10k images
│   └── split_kadid10k.csv
├── koniq10k/
│   ├── 512x384/          # KonIQ-10k images (comes with own split)
│   └── koniq10k_*.csv    # Original split files
└── AGIQA-3K/
    ├── images/           # All AGIQA-3k images
    └── split_agiqa3k.csv

Important Notes:

• KADID-10k: Move split_kadid10k.csv into the kadid10k/ folder
• KonIQ-10k: Uses its own original split files, no need to move
• AGIQA-3k: Move split_agiqa3k.csv into the AGIQA-3K/ folder; images are in the images/ subfolder

Pre-trained Checkpoints

We provide pre-trained visual prompt checkpoints on HuggingFace Hub for immediate use:

🔗 Download Checkpoints

The checkpoints are provided as visual_prompt_ckpt_trained_on_mplug2.zip containing training experiment folders with checkpoint directories (checkpoint-xxxx). Each experiment folder contains multiple epochs, and the best performing checkpoint can be identified from the best_model_checkpoint info in the final checkpoint folder.

To use the pre-trained checkpoints:

1. Download and extract the checkpoint archive:

   # Download from HuggingFace Hub
   wget https://huggingface.co/yahya007/mplug2-vp-for-nriqa/blob/main/visual_prompt_ckpt_trained_on_mplug2.zip
   unzip visual_prompt_ckpt_trained_on_mplug2.zip
   

2. Navigate to the desired experiment folder:

   cd SGD_mplug2_exp_04_kadid_padding_30px_add/
   # Check the latest checkpoint folder (highest number)
   ls -la checkpoint-*/
   # Look for best_model_checkpoint info in the final checkpoint
   

3. Update the configuration and checkpoint in src/tester.py:

   # Update config path
   config_path = "configs/final_mplug_owl2_configs/SGD_mplug2_exp_04_kadid_padding_30px_add.yaml"
   
   # Update checkpoint name - use "checkpoint-best" or specific checkpoint number
   checkpoint_best = "checkpoint-best"  # or "checkpoint-XXXX" for specific epoch
   

4. Run inference:

   cd src
   python tester.py
   

The inference script outputs:

• SRCC (Spearman Rank Correlation Coefficient)
• PLCC (Pearson Linear Correlation Coefficient)

📈 Results

Best Performance (30px Padding + Addition)

Performance Across Visual Prompt Types

Comparison with State-of-the-Art Methods

Comparison with Specialized NR-IQA Models

✍️ Citation

If you use this work, please cite our paper:

@article{benmahaneHassouni2025mplugvpiqa,
  title        = {Parameter-Efficient Adaptation of mPLUG-Owl2 via Pixel-Level Visual Prompts for NR-IQA},
  author       = {Benmahane, Yahya and El Hassouni, Mohammed},
  journal      = {arXiv preprint arXiv:2509.03494},
  year         = {2025},
  url          = {https://arxiv.org/abs/2509.03494}
}

📚 Acknowledgments

• mPLUG-Owl2 for the base multimodal LLM
• HuggingFace Transformers for the training framework
• Bahng et al. (2022)