MedCLIPSeg

MedCLIPSeg: Probabilistic Vision–Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation

<a href="https://arxiv.org/abs/2602.20423" target="_blank"><img alt="arXiv" src="https://img.shields.io/badge/arXiv-2602.20423-B31B1B?logo=arxiv&logoColor=white" height="25"/></a> <a href="https://tahakoleilat.github.io/MedCLIPSeg" target="_blank"><img alt="Project Website" src="https://img.shields.io/badge/%F0%9F%94%97%20Project-Website-blue" height="25"/></a> <a href="https://github.com/HealthX-Lab/MedCLIPSeg" target="_blank"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-1f6feb?logo=github&logoColor=white" height="25"/></a> <a href="https://huggingface.co/datasets/TahaKoleilat/MedCLIPSeg" target="_blank"><img alt="HuggingFace Dataset" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Datasets-Access-ffc107?color=ffc107&logoColor=white" height="25"/></a> <a href="https://huggingface.co/TahaKoleilat/MedCLIPSeg" target="_blank"><img alt="HuggingFace Models" src="https://img.shields.io/badge/Models-Reproduce-2ea44f?logo=huggingface&logoColor=white" height="25"/></a> <a href="#citation"><img alt="Citation" src="https://img.shields.io/badge/Citation-BibTeX-6C63FF?logo=bookstack&logoColor=white" height="25"/></a>

This repository hosts the official trained model checkpoints for MedCLIPSeg, presented in the paper MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation.

Authors: Taha Koleilat, Hojat Asgariandehkordi, Omid Nejati Manzari, Berardino Barile, Yiming Xiao, Hassan Rivaz.

MedCLIPSeg is a vision–language framework for medical image segmentation built on top of CLIP. It adapts CLIP for robust, data-efficient, and uncertainty-aware segmentation through probabilistic cross-modal attention and bidirectional interaction between image and text tokens.

The released checkpoints correspond exactly to the experiments reported in our paper and are provided for evaluation and reproducibility purposes only.

🧠 Model Overview

• Backbone: UniMedCLIP ViT-B/16
• Task: Medical Image Segmentation
• Modalities: Ultrasound, MRI, Endoscopy, Dermoscopy, X-ray
• Training Regimes:
  • Data Efficiency Evaluation
  • Fully supervised learning
  • Domain generalization

🚀 Reproducing Paper Results

Step 1: Download the Checkpoints

Download the checkpoints here, then create a directory named outputs_medclipseg at the root of the project and place the downloaded checkpoint folders inside it so that the directory structure matches the following layout:

outputs_medclipseg/
├── BUSI/
├── BTMRI/
├── ISIC/
├── Kvasir/
├── Covid19/
├── EUS/
└── ...

Each folder contains the trained UniMedCLIP-based MedCLIPSeg checkpoints for that dataset.

Step 2: Run Evaluation

Run the following script to reproduce the results reported in the paper:

bash scripts/reproduce_eval.sh

This script automatically loads the corresponding checkpoints and evaluates them on the appropriate test sets.

📊 Outputs

Evaluation outputs (segmentations and uncertainty maps) are written to:

outputs_medclipseg/<DATASET>/seg_results/
outputs_medclipseg/<DATASET>/unc_results/

📚 Acknowledgment of Foundation Models

The underlying vision–language models used in this repository were introduced in prior work. We gratefully acknowledge the original authors:

• PubMedCLIP
  Eslami, Sedigheh, Gerard De Melo, and Christoph Meinel. "Does clip benefit visual question answering in the medical domain as much as it does in the general domain?." arXiv preprint arXiv:2112.13906 (2021).

• UniMedCLIP
  Khattak, Muhammad Uzair, et al. "Unimed-clip: Towards a unified image-text pretraining paradigm for diverse medical imaging modalities." arXiv preprint arXiv:2412.10372 (2024).

For completeness and reproducibility, this repository also includes the original pretrained checkpoints of these foundation models under the checkpoints/ directory, exactly as released by their respective authors.

All MedCLIPSeg checkpoints are adaptations built on top of these pretrained models and are released strictly for research and non-commercial use, in accordance with their respective licenses.

📖 Citation

If you use these checkpoints in your research, please cite:

@article{koleilat2026medclipseg,
  title={MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation},
  author={Koleilat, Taha and Asgariandehkordi, Hojat and Manzari, Omid Nejati and Barile, Berardino and Xiao, Yiming and Rivaz, Hassan},
  journal={arXiv preprint arXiv:2602.20423},
  year={2026}
}