Adapt-MLLM-to-Domains

Adapting Multimodal Large Language Models to Domains via Post-Training (EMNLP 2025)

This project adapts general Multimodal Large Language Models (MLLMs) to specific domains like science and industry to improve their real-world use. It focuses on three main areas:

1. Data Synthesis

• We create a generate-then-filter pipeline using open-source models to make diverse visual tasks from domain-specific image-caption pairs.
• This data works better than data made by hand or closed-source models (e.g., GPT-4V/o).

2. Training Pipeline

• Instead of the usual two-step training (image-caption pairs first, then visual tasks), we use a single-stage training to handle more tasks for specific domains.

3. Task Evaluation

• We test our method in high-impact domains like biomedicine, food, and remote sensing.
• We train and evaluate MLLMs on domain-specific tasks to show how well they perform.

Resources

🤗 We share our data and models with example usages, feel free to open any issues or discussions! 🤗

Code: https://github.com/bigai-ai/QA-Synthesizer

Citation

If you find our work helpful, please cite us.

Adapt MLLM to Domains (EMNLP 2025 Findings)

@article{adamllm,
  title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
  author={Cheng, Daixuan and Huang, Shaohan and Zhu, Ziyu and Zhang, Xintong and Zhao, Wayne Xin and Luan, Zhongzhi and Dai, Bo and Zhang, Zhenliang},
  journal={arXiv preprint arXiv:2411.19930},
  year={2024}
}

Adapt LLM to Domains (ICLR 2024)

@inproceedings{
adaptllm,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}