open-pmc-clip

<div align="center"> <img src="https://github.com/VectorInstitute/pmc-data-extraction/blob/0a969136344a07267bb558d01f3fe76b36b93e1a/media/open-pmc-pipeline.png?raw=true" alt="Open-PMC Pipeline" width="1000" /> </div> <p align="center"> <strong>Arxiv:</strong> <a href="http://arxiv.org/abs/2503.14377" target="_blank">Arxiv</a> &nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;&nbsp; <strong>Code:</strong> <a href="https://github.com/VectorInstitute/pmc-data-extraction" target="_blank">Open-PMC Github</a> &nbsp;&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;&nbsp; <strong>Dataset:</strong> <a href="https://huggingface.co/datasets/vector-institute/open-pmc" target="_blank">Hugging Face</a> </p>

Model Overview

This model is a checkpoint trained on the Open-PMC dataset. It utilizes a Vision Transformer (ViT-base16) as the backbone for visual feature extraction and PubMedBERT for processing text data. The model is trained using Contrastive Learning with the vanilla Info-NCE loss to learn meaningful representations across different modalities.

Model Architecture

• Vision Backbone: ViT-B/16 (Pretrained on ImageNet)
• Text Backbone: PubMedBERT (Pretrained on PubMedCentral Abstracts)
• Training Objective: Contrastive Learning with Info-NCE Loss

Training Framework

The model was trained using the mmlearn framework, which is designed for multimodal learning. You can find more information and access the framework here.

How to Use

Please visit out GitHub for information on how to run benchmarking using this checkpoint