biomed-Llama-3.2-11B-Vision-Instruct

Adapting Multimodal Large Language Models to Domains via Post-Training (EMNLP 2025)

This repos contains the biomedicine MLLM developed from Llama-3.2-11B-Vision-Instruct in our paper: On Domain-Specific Post-Training for Multimodal Large Language Models. The correspoding training dataset is in medicine-visual-instructions.

The main project page is: Adapt-MLLM-to-Domains

1. To Chat with AdaMLLM

Our model architecture aligns with the base model: Llama-3.2-Vision-Instruct. We provide a usage example below, and you may refer to the official Llama-3.2-Vision-Instruct Repository for more advanced usage instructions,

Note: For AdaMLLM, always place the image at the beginning of the input instruction in the messages.

<details> <summary> Click to expand </summary>

Starting with transformers >= 4.45.0 onward, you can run inference using conversational messages that may include an image you can query about.

Make sure to update your transformers installation via pip install --upgrade transformers.

import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor

model_id = "AdaptLLM/biomed-Llama-3.2-11B-Vision-Instruct"

model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# NOTE: For AdaMLLM, always place the image at the beginning of the input instruction in the messages.  
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
    ]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))

</details>

2. To Evaluate Any MLLM on Domain-Specific Benchmarks

See biomed-VQA-benchmark to reproduce our results and evalaute more MLLMs on the domain-specific benchmarks.

3. To Reproduce this Domain-Adapted MLLM

See Post-Train Guide to adapt MLLMs to domains.

Citation

If you find our work helpful, please cite us.

Adapt MLLM to Domains (EMNLP 2025 Findings)

@article{adamllm,
  title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
  author={Cheng, Daixuan and Huang, Shaohan and Zhu, Ziyu and Zhang, Xintong and Zhao, Wayne Xin and Luan, Zhongzhi and Dai, Bo and Zhang, Zhenliang},
  journal={arXiv preprint arXiv:2411.19930},
  year={2024}
}

Adapt LLM to Domains (ICLR 2024)

@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}