pubmedbert-chia-ner

PubMedBERT/BiomedBERT fine-tuned for CHIA clinical trial eligibility NER

Fine-tuned microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext for named entity recognition on clinical trial eligibility criteria (CHIA corpus), as part of a course NLP project comparing fine-tuned biomedical transformers vs. GPT-4 prompting.

• Code / full writeup: https://github.com/jatinpsingh/NER_Clinical_Trial_Eligibility/tree/main/pmb
• Base model: PubMedBERT-base (renamed BiomedBERT), Gu et al. 2021
• Task: token classification, 31 labels (O + 15 entity types x B/I)
• Data: CHIA eligibility criteria, team's shared fixed split (10,006 train / 1,240 val / 1,163 test sentences -- fold 0 of a 10-fold CV scheme, see below)

Hyperparameters (matching Li et al. 2022, Table 3)

Results

This checkpoint's own fold (entity-level, test set):

Full 10-fold cross-validation (mean +/- std across all 10 folds -- the number directly comparable to Li et al. 2022's own 10-fold-averaged reporting; the weights hosted in this repo are from one of these 10 folds, not a checkpoint averaged across them):

Both this checkpoint's own score and the 10-fold mean exceed Li et al. 2022's published PubMedBERT numbers on Chia (0.622 strict / 0.744 relaxed).

Usage

from transformers import AutoModelForTokenClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ptanwar/pubmedbert-chia-ner")
model = AutoModelForTokenClassification.from_pretrained("ptanwar/pubmedbert-chia-ner")

Known limitations

• The hosted weights are one fold's model, not a checkpoint averaged/ensembled across the 10-fold CV run -- treat the single-fold numbers above as this specific checkpoint's performance, and the 10-fold mean +/- std as the more rigorous estimate of the approach's true performance.
• Weakest on rare/ambiguous types (Mood, Observation, Reference_point) -- see the full writeup for error analysis.