Merlin-Cancer-Super

<p align="center"> <img src="https://huggingface.co/datasets/AbdomenAtlas/MerlinPlus/resolve/main/documents/tumor_examples.png" alt="Examples of tumor annotations in Merlin Plus" width="100%"> </p>

Merlin-Cancer-Super

Report Supervision (R-Super) is an AI training method that learns tumor detection and segmentation from radiology reports (MICCAI 2025, Best Paper Award, top 2). Merlin-Cancer-Super is R-Super trained on the radiology reports and tumor masks from the Merlin Plus dataset (MICCAI 2026). It surpasses state-of-the-art public AI models and leading VLMs in the detection and segmentation of tumors in 9 organs: bladder, gallbladder, spleen, esophagus, stomach, duodenum, prostate, uterus, and adrenal glands.

Report Supervision. R-Super is a training strategy that transforms radiology reports (text) into direct (per-voxel) supervision for tumor segmentation AI. Before training, we use an LLM to extract tumor information from radiology reports. Then, R-Super introduces new loss functions (Volume Loss & Ball Loss), which use this extracted information to teach the AI to segment tumors that are coherent with the reports, in terms of tumor count, diameters, and locations. GitHub: https://github.com/MrGiovanni/R-Super

Merlin Plus is a dataset with longitudinal metadata (patient IDs and scan dates) and per-voxel annotations for organs and 9 tumor types in the Merlin dataset (Stanford, 25,494 CT scans). Merlin Plus is part of a collaboration between the Merlin Project at Stanford and the R-Super Project at Johns Hopkins University. It is available at HuggingFace: https://huggingface.co/datasets/AbdomenAtlas/MerlinPlus

Performance vs. Previous Public AI Models

Results from the Merlin Plus paper (MICCAI 2026, click for details).

<p align="center"> <img src="https://github.com/MrGiovanni/MerlinPlus/blob/main/document/performance_public_models.png?raw=true" alt="Performance" width="100%"> </p>

Papers

<b>Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks</b> <br/> Pedro R. A. S. Bassi, Xinze Zhou, Wenxuan Li, Szymon Płotka, Jieneng Chen, Qi Chen, Zheren Zhu, Jakub Prządo, Ibrahim E. Hamacı, Sezgin Er, Yuhan Wang, Ashwin Kumar, Bjoern Menze, Jarosław B. Ćwikła, Yuyin Zhou, Akshay S. Chaudhari, Curtis P. Langlotz, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan L. Yuille, Zongwei Zhou* <br/> Johns Hopkins University, Stanford University, University of Zurich, University of Warsaw, Italian Institute of Technology, UCSF <br/> Preprint <br/> <a href='https://www.cs.jhu.edu/~zongwei/publication/bassi2025scaling.pdf'><img src='https://img.shields.io/badge/Paper-PDF-purple'></a> <a href="https://github.com/MrGiovanni/R-Super"> <img src="https://img.shields.io/badge/GitHub-RSuper-black?logo=github"> </a>

<b>Learning Segmentation from Radiology Reports</b> <br/> Pedro R. A. S. Bassi, Wenxuan Li, Jieneng Chen, Zheren Zhu, Tianyu Lin, Sergio Decherchi, Andrea Cavalli, Kang Wang, Yang Yang, Alan Yuille, Zongwei Zhou* <br/> Johns Hopkins University <br/> MICCAI 2025 <br/> <b>Best Paper Award Runner-up (top 2 in 1,027 papers)</b> <br/> <a href='https://link.springer.com/chapter/10.1007/978-3-032-04971-1_29'><img src='https://img.shields.io/badge/Paper-PDF-purple'></a><a href='https://link.springer.com/chapter/10.1007/978-3-032-04971-1_29'><img src='https://img.shields.io/badge/Springer-Link-orange'></a> <a href="https://github.com/MrGiovanni/R-Super"> <img src="https://img.shields.io/badge/GitHub-RSuper-black?logo=github"> </a>

Merlin Plus: A Large-Scale, Multi-cancer, Image-Mask-Report Dataset
Pedro R. A. S. Bassi†, Wenxuan Li†, Szymon Płotka†, Ruby Honjol, Jakub Prządo, Xinze Zhou, Kang Wang, Yang Yang, Malte Jensen, Akshay S. Chaudhari, Curtis P. Langlotz, Alan L. Yuille, and Zongwei Zhou.
MICCAI 2026, LNCS 16895. Springer Nature Switzerland.
<a href="https://papers.miccai.org/miccai-2026/paper/4063_paper.pdf"><img src="https://img.shields.io/badge/Paper-PDF-purple" alt="Paper PDF"></a> <a href="https://huggingface.co/datasets/AbdomenAtlas/MerlinPlus"> <img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Dataset-yellow"> </a> <a href="https://stanford.redivis.com/datasets/bvj1-5hc2whwf6"> <img src="https://img.shields.io/badge/Stanford%20AIMI-Dataset-8C1515"> </a> <a href="https://github.com/MrGiovanni/MerlinPlus"> <img src="https://img.shields.io/badge/GitHub-MerlinPlus-black?logo=github"> </a>

<b>Merlin: A Vision Language Foundation Model for 3D Computed Tomography</b> <br/> Louis Blankemeier, Joseph Paul Cohen, Ashwin Kumar, Dave Van Veen, Syed Jamal Safdar Gardezi, Magdalini Paschali, Zhihong Chen, Jean-Benoit Delbrouck, Eduardo Reis, Cesar Truyts, Christian Bluethgen, Malte Engmann Kjeldskov Jensen, Sophie Ostmeier, Maya Varma, Jeya Maria Jose Valanarasu, Zhongnan Fang, Zepeng Huo, Zaid Nabulsi, Diego Ardila, Wei-Hung Weng, Edson Amaro Junior, Neera Ahuja, Jason Fries, Nigam H. Shah, Andrew Johnston, Robert D. Boutin, Andrew Wentland, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, Akshay S. Chaudhari <br/> Stanford University <br/> Nature, 2026. <br/> <a href='https://www.nature.com/articles/s41586-026-10181-8'><img src='https://img.shields.io/badge/Paper-PDF-purple'></a> <a href="https://github.com/StanfordMIMI/Merlin"> <img src="https://img.shields.io/badge/GitHub-Merlin-black?logo=github"> </a>

Instructions

Data format

The code expects data in the format below. If your data is not in this format, see the format conversion instructions below.

/path/to/dataset/
├── BDMAP_00000001/
|    └── ct.nii.gz
├── BDMAP_00000002/
|    └── ct.nii.gz
...

We also need a csv file with the cases to predict (the names of the files above), in a BDMAP ID column:

BDMAP ID
BDMAP_00000001
BDMAP_00000002
...

Format conversion and csv creation:

You can use the code below to convert your dataset to our input format and create the csv. This conversion script assumes that your data is organized as nii.gz files inside a single folder (/path/to/flat/). The csv will be saved to ids.csv

python convert_to_abdomenatlas.py \
    --input_dir /path/to/flat/ \
    --output_dir /path/to/dataset/

Installation

git clone https://github.com/PedroRASB/SegmentationFromReports
cd SegmentationFromReports/rsuper_train
conda create -n rsuper python=3.10 -y
conda activate rsuper
pip install -r requirements.txt

Inference

python predict_abdomenatlas.py \
    --load ../../checkpoint/fold_0_latest.pth \
    --class_list ../../checkpoint/label_names.yaml \
    --img_path /path/to/dataset/ \
    --ids /path/to/ids.csv \
    --save_path /path/to/outputs/ \
    --gpu 0

Outputs one folder per case, with a mask for each of the 9 tumors and 26 organs:

/path/to/outputs/abdomenatlas/checkpoint/
├── BDMAP_00000001/
|    └── predictions/
|         ├── spleen_lesion.nii.gz
|         ├── stomach_lesion.nii.gz
|         ├── liver.nii.gz
|         └── ...

Add --save_probabilities_lesions to also write the tumor probability maps to predictions_raw/. To use several GPUs, split the case list:

for p in 0 1 2 3; do
  python predict_abdomenatlas.py --load ... --class_list ... --img_path ... \
      --ids ... --save_path ... --gpu $p --parts 4 --current_part $p &
done

Citations

@article{bassi2025scaling,
  title={Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks},
  author={Bassi, Pedro R. A. S. and Zhou, Xinze and Li, Wenxuan and P{\l}otka, Szymon and Chen, Jieneng and Chen, Qi and Zhu, Zheren and Prz{\k{a}}do, Jakub and Hamamci, Ibrahim E. and Er, Sezgin and others},
  journal={arXiv preprint arXiv:2510.14803},
  year={2025}
}

@inproceedings{bassi2025learning,
  title={Learning Segmentation from Radiology Reports},
  author={Bassi, Pedro R. A. S. and Li, Wenxuan and Chen, Jieneng and Zhu, Zheren and Lin, Tianyu and Decherchi, Sergio and Cavalli, Andrea and Wang, Kang and Yang, Yang and Yuille, Alan L. and Zhou, Zongwei},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={305--315},
  year={2025},
  publisher={Springer},
  doi={10.1007/978-3-032-04971-1_29}
}


@InProceedings{BasPed_Merlin_MICCAI2026,
    author = { Bassi, Pedro R. A. S. AND Li, Wenxuan AND Płotka, Szymon AND Honjol, Ruby AND Prządo, Jakub AND Zhou, Xinze AND Wang, Kang AND Yang, Yang AND Jensen, Malte AND Chaudhari, Akshay S. AND Langlotz, Curtis P. AND Yuille, Alan L. AND Zhou, Zongwei},
    title = { { Merlin Plus: A Large-Scale, Multi-cancer, Image-Mask-Report Dataset } },
    booktitle = {Medical Image Computing and Computer Assisted Intervention -- MICCAI 2026},
    year = {2026},
    publisher = {Springer Nature Switzerland},
    volume = {LNCS 16895},
    month = {September},
    page = {pending}
}

@article{blankemeier_kumar2026merlin,
  author = {Blankemeier, Louis and Kumar, Ashwin and Cohen, Joseph Paul and Liu, Jiaming and Liu, Longchao and Van Veen, Dave and Gardezi, Syed Jamal Safdar and Yu, Hongkun and Paschali, Magdalini and Chen, Zhihong and Delbrouck, Jean-Benoit and Reis, Eduardo and Holland, Robbie and Truyts, Cesar and Bluethgen, Christian and Wu, Yufu and Lian, Long and Jensen, Malte Engmann Kjeldskov and Ostmeier, Sophie and Varma, Maya and Valanarasu, Jeya Maria Jose and Fang, Zhongnan and Huo, Zepeng and Nabulsi, Zaid and Ardila, Diego and Weng, Wei-Hung and Amaro Junior, Edson and Ahuja, Neera and Fries, Jason and Shah, Nigam H. and Zaharchuk, Greg and Willis, Marc and Yala, Adam and Johnston, Andrew and Boutin, Robert D. and Wentland, Andrew and Langlotz, Curtis P. and Hom, Jason and Gatidis, Sergios and Chaudhari, Akshay S.},
  title   = {Merlin: a computed tomography vision-language foundation model and dataset},
  journal = {Nature},
  year    = {2026},
  doi     = {10.1038/s41586-026-10181-8},
  url     = {https://doi.org/10.1038/s41586-026-10181-8}
}

Acknowledgement

This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research, the Patrick J. McGovern Foundation Award, and the National Institutes of Health (NIH) under Award Number R01EB037669. Paper content is covered by patents pending.

© The Johns Hopkins University. This work is openly licensed via CC BY-NC-ND. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License