medpmc-caption-separation-internvl-2.5-4b-mpo

MedPMC Caption Separation Model

This model separates a compound figure caption into subcaptions corresponding to its ordered subfigures.

Given a compound figure, its subfigures, and the original main caption, the model extracts one subcaption for each subfigure. Subcaptions are returned in the same order as the input subfigures and separated using ||.

The model was presented in the paper MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models.

Task

Input

• One compound figure
• Multiple ordered subfigures
• One main caption

Output

• Subcaptions separated by ||
• The number and order of output subcaptions should match the input subfigures

Prompt Format

Your task is to separate the given caption into subcaptions. You are provided with a compound figure, {N} subfigures, and a main caption. For each subfigure, extract the corresponding subcaption from the main caption and separate them using "||". Make sure the number and order of subcaptions match the given subfigures.

# Compound Figure
<image>
# Subfigure
<image>
# Subfigure
<image>
...
# Main Caption
{main_caption}

Example

Input

Your task is to separate the given caption into subcaptions. You are provided with a compound figure, 2 subfigures, and a main caption. For each subfigure, extract the corresponding subcaption from the main caption and separate them using "||". Make sure the number and order of subcaptions match the given subfigures.

# Compound Figure
<image>
# Subfigure
<image>
# Subfigure
<image>
# Main Caption
Figure 3. Effects of LIPUS treatment on brain edema in TBI mice. (a) Representative T2-weighted MRI images at 1 and 4 days post-TBI. The damaged area is defined as a hyperintense region over the right parietal cortex, indicating edema formation. Dotted line shows location of regions of interest. (b) Quantification revealed significantly smaller edema volumes in LIPUS-treated mice compared with non-treated mice at 1 and 4 days. # Denotes significantly different from non-treated TBI group (### p < 0.001, n = 6).

Output

( a ) Representative T2-weighted MRI images at 1 and 4 days post - TBI . The damaged area is defined as a hyperintense region over the right parietal cortex , indicating edema formation . Dotted line shows location of regions of interest .||( b ) Quantification revealed significantly smaller edema volumes in LIPUS - treated mice compared with non - treated mice at 1 and 4 days .

Requirements

The examples below were tested with LMDeploy 0.14.0.

pip install "lmdeploy==0.14.0" timm pillow

Inference with LMDeploy

from lmdeploy import TurbomindEngineConfig, pipeline
from lmdeploy.vl import load_image
from lmdeploy.vl.constants import IMAGE_TOKEN


MODEL_PATH = (
    "Yale-BIDS-Chen/"
    "medpmc-caption-separation-internvl-2.5-4b-mpo"
)


def build_prompt(main_caption: str, num_subfigures: int) -> str:
    subfigure_blocks = "\n".join(
        f"# Subfigure\n{IMAGE_TOKEN}"
        for _ in range(num_subfigures)
    )

    return (
        "Your task is to separate the given caption into subcaptions. "
        f"You are provided with a compound figure, {num_subfigures} "
        "subfigures, and a main caption. "
        "For each subfigure, extract the corresponding subcaption from "
        "the main caption and separate them using \"||\". "
        "Make sure the number and order of subcaptions match the given "
        "subfigures.\n\n"
        f"# Compound Figure\n{IMAGE_TOKEN}\n"
        f"{subfigure_blocks}\n"
        "# Main Caption\n"
        f"{main_caption}"
    )


pipe = pipeline(
    MODEL_PATH,
    backend_config=TurbomindEngineConfig(session_len=32768),
    trust_remote_code=True,
)

image_paths = [
    "compound_figure.png",
    "subfigure_1.png",
    "subfigure_2.png",
]

main_caption = (
    "Figure 3. Effects of treatment on the measured outcome. "
    "(a) Representative images from the control and treatment groups. "
    "(b) Quantification of the outcome across groups."
)

prompt = build_prompt(
    main_caption=main_caption,
    num_subfigures=len(image_paths) - 1,
)

images = [load_image(path) for path in image_paths]
response = pipe((prompt, images))

subcaptions = [
    text.strip()
    for text in response.text.split("||")
]

print(response.text)
print(subcaptions)

Batch Inference

The following example assumes a JSONL file in the LLaVA/InternVL conversation format. The first image is the compound figure, and the remaining images are ordered subfigures.

import json
import os
from pathlib import Path

from lmdeploy import TurbomindEngineConfig, pipeline
from lmdeploy.vl import load_image
from lmdeploy.vl.constants import IMAGE_TOKEN


def run_inference(
    model_path: str,
    input_jsonl: str,
    image_root: str,
    output_json: str,
    batch_size: int = 4,
) -> None:
    pipe = pipeline(
        model_path,
        backend_config=TurbomindEngineConfig(session_len=32768),
        trust_remote_code=True,
    )

    with open(input_jsonl, encoding="utf-8") as file:
        examples = [
            json.loads(line)
            for line in file
            if line.strip()
        ]

    outputs = []

    for start in range(0, len(examples), batch_size):
        batch = examples[start:start + batch_size]
        requests = []

        for example in batch:
            image_paths = [
                os.path.join(image_root, filename)
                for filename in example["image"]
            ]

            images = [
                load_image(path)
                for path in image_paths
            ]

            prompt = example["conversations"][0]["value"].replace(
                "<image>",
                IMAGE_TOKEN,
            )

            requests.append((prompt, images))

        responses = pipe(requests)

        for example, response in zip(batch, responses):
            subcaptions = [
                text.strip()
                for text in response.text.split("||")
            ]

            expected_count = len(example["image"]) - 1

            outputs.append(
                {
                    "id": example.get("id"),
                    "prediction": response.text,
                    "prediction_subcaptions": subcaptions,
                    "expected_subcaption_count": expected_count,
                    "predicted_subcaption_count": len(subcaptions),
                    "finish_reason": getattr(
                        response,
                        "finish_reason",
                        None,
                    ),
                    "valid": (
                        len(subcaptions) == expected_count
                        and all(subcaptions)
                        and getattr(response, "finish_reason", None)
                        != "length"
                    ),
                }
            )

    output_path = Path(output_json)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    with output_path.open("w", encoding="utf-8") as file:
        json.dump(
            outputs,
            file,
            indent=2,
            ensure_ascii=False,
        )


if __name__ == "__main__":
    run_inference(
        model_path=(
            "Yale-BIDS-Chen/"
            "medpmc-caption-separation-internvl-2.5-4b-mpo"
        ),
        input_jsonl="input.jsonl",
        image_root="images",
        output_json="outputs/predictions.json",
        batch_size=4,
    )

Output Format

The model returns a single string:

{subcaption_1}||{subcaption_2}||...||{subcaption_N}

It can be parsed using:

subcaptions = [
    text.strip()
    for text in response.text.split("||")
]

Notes

• The first image must be the original compound figure.
• The remaining images must be the subfigures in their target order.
• The number and order of output subcaptions should match the input subfigures.
• For MedPMC data curation, an example is retained only when: (1) The number of predicted subcaptions equals the number of input subfigures and (2) Every predicted subcaption is non-empty.
• In the generated output, global caption information not associated with a specific subfigure may be omitted.
• Model outputs may contain tokenization-style spacing, such as ( a ) or post - TBI. Optional spacing normalization may be applied after parsing and validation.

License

The model is released for non-commercial research use under CC BY-NC-SA 4.0.

Citation

@article{kim2026medpmc,
  title={MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models},
  author={Kim, Hyunjae and Kim, Dain and Xiao, Pan and Applebaum, Serina S and Chung, Younjoon and Ai, Xuguang and Yin, Yu and Jiang, Roy and Du, Yuexi and Wei, Yawen and others},
  journal={arXiv preprint arXiv:2607.07673},
  year={2026}
}

Questions?

For questions or feedback, please contact Hyunjae Kim at [email protected].