ModernPubMedBERT

Clinical ModernBERT Embedding Model

A specialized medical embedding model fine-tuned from Clinical ModernBERT using InfoNCE contrastive learning on PubMed title-abstract pairs.

Model Details

• Base Model: thomas-sounack/BioClinical-ModernBERT-base
• Training Method: InfoNCE contrastive learning
• Training Data: PubMed title-abstract pairs
• Max Sequence Length: 2048 tokens

Usage

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("lokeshch19/ModernPubMedBERT")

# Encode medical texts
texts = [
    "Rheumatoid arthritis is an autoimmune disorder attacking joint linings.",
    "Inflammatory cytokines in RA lead to progressive cartilage and bone destruction."
]
embeddings = model.encode(texts)

Applications

• Medical document similarity analysis
• Clinical text retrieval systems
• Biomedical literature search
• Medical concept matching and classification

Model Comparison

Compared to NeuML/bioclinical-modernbert-base-embeddings, our model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content.

Comprehensive Evaluation Results

Evaluation performed using InformationRetrievalEvaluator from sentence-transformers on the gamino/wiki_medical_terms dataset.

Model Comparison

Compared to NeuML/bioclinical-modernbert-base-embeddings, this model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content.

Medical Text Similarity

Example 1: Related Medical Concepts

text1 = "Hypertension increases the risk of stroke and heart attack."
text2 = "High blood pressure damages arterial walls over time, leading to cardiovascular events."

# Cosine Similarity Results:
# Our Model: 0.5941 (59.4%)
# NeuML Model: 0.5267 (52.7%)
# Improvement: +12.7%

Non-Medical Text Discrimination

Example 2: Medical vs. Programming Terms

texts = ["diabetes type 2", "asyncio.run()"]

# Cosine Similarity Results:
# Our Model: 0.0804 (8.0%) - Correctly identifies low similarity
# NeuML Model: 0.1926 (19.3%) - Higher false similarity
# Better Discrimination: 58% lower false positive rate

Key Advantages

• Enhanced Medical Understanding: 12.7% better similarity detection for related medical concepts
• Improved Discrimination: 58% reduction in false similarities between medical and non-medical terms
• Domain Specialization: Fine-tuned specifically on PubMed literature for optimal medical text processing

Training Details

• Optimizer: AdamW (learning rate: 3e-4, weight decay: 0.1)
• Batch Size: 72
• Training Steps: 7,000
• Warmup Steps: 700

Citation

If you use this model, please cite the base model paper and acknowledge this fine-tuning work.