Clinical_ModernBERT

Clinical ModernBERT

Clinical ModernBERT is a state-of-the-art encoder-based transformer tailored specifically for biomedical and clinical text handling context length up to 8192 tokens. Building on the innovations introduced by ModernBERT, this model extends the context window to 8,192 tokens and incorporates domain-specific vocabulary refinements. It is designed to produce semantically rich representations that capture both the nuanced syntax of biomedical literature and the intricate semantics of clinical narratives.

Usage

Pretrained model weights and tokenizer artifacts are provided to facilitate easy integration with your downstream biomedical NLP tasks:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('Simonlee711/Clinical_ModernBERT')
tokenizer = AutoTokenizer.from_pretrained('Simonlee711/Clinical_ModernBERT')

Model Overview

Below is a table summarizing ModernBERT's key architectural components and their benefits:

This model leverages a suite of modern architectural advancements including rotary positional embeddings (RoPE), Flash Attention for near-linear memory usage with extended contexts, and GeGLU activation layers that enhance representational capacity by integrating smooth gating mechanisms. By initializing from a ModernBERT-base checkpoint and applying domain-specific pre-training on approximately 40 million PubMed abstracts combined with MIMIC-IV clinical notes, Clinical ModernBERT is optimized to serve in tasks such as retrieval-augmented generation, fine-grained text classification, and domain-specific entity extraction.

Pre-training Optimizations

Masked Language Modeling (MLM) Setup

Clinical ModernBERT is pre-trained using a multi-phase masked language modeling (MLM) strategy. A custom collator dynamically adjusts the masking probability—beginning at 30% and decreasing to 15% over the course of training—to emphasize medically relevant tokens (e.g., drug names, procedural codes). The MLM objective is defined as

$$ \mathcal{L}{\text{MLM}} = - \sum{i \in \mathcal{M}} \log p_\theta(x_i \mid \tilde{x}), $$

where the model predicts masked tokens given their context. The table below summarizes the MLM evaluation framework without detailing specific metric scores:

This structure captures the granularity at which the model’s recovery of masked tokens is evaluated, with the understanding that higher top-(k) values reflect a broader lexical recall, and the model consistently ranks clinically appropriate tokens among its predictions.

Intended Use

Clinical ModernBERT is ideally suited for tasks that demand an in-depth understanding of biomedical language. It is particularly valuable for clinical information retrieval, narrative classification, and structured medical coding. Researchers and practitioners may fine-tune this model for specialized downstream applications such as electronic health record analysis, clinical decision support systems, and evidence-based medical literature retrieval.

Citations and Pre-training Source Code

The source code can be found here: Clinical ModernBERT Github

Citing Model

@misc{simon_lee_2025,
	author       = { Simon A. Lee },
	title        = { Clinical_ModernBERT (Revision 24e72d6) },
	year         = 2025,
	url          = { https://huggingface.co/Simonlee711/Clinical_ModernBERT },
	doi          = { 10.57967/hf/4999 },
	publisher    = { Hugging Face }
}

Citing Paper

@article{lee2025clinical,
  title={Clinical ModernBERT: An efficient and long context encoder for biomedical text},
  author={Lee, Simon A and Wu, Anthony and Chiang, Jeffrey N},
  journal={arXiv preprint arXiv:2504.03964},
  year={2025}
}

Questions

email ([email protected])