im-bin-tf-titles

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

im-bin-tf-abstr

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1908
• Accuracy: 0.9222
• F1: 0.9220
• Precision: 0.9267
• Recall: 0.9174
• Roc Auc: 0.9781

Model description

im-bin-tf-abstr is a fine-tuned [PubMedBERT](https://huggingface.co/microsoft/BiomedNLP-PubMedB ERT-base-uncased-abstract-fulltext)
model for binary classification of biomedical text as Internal Medicine (IM) or Other.
It was trained on 300,000 PubMed article titles using journal provenance as a distant
supervision signal: titles from journals covering any of the eleven IM sub-specialties
recognised by the German Society of Internal Medicine (DGIM) were labeled IM; titles from
six other disciplines (anesthesiology, otorhinolaryngology, gynecology, surgery, psychiatry,
neurology) were labeled Other. No manual annotation was used.

The model was developed as part of a dissertation at the DGIM and evaluated in a prospective
human study at the 129th Congress of Internal Medicine (Wiesbaden, 2023), where it
significantly outperformed board-certified internists on the same classification task.

Intended uses & limitations

• Automatic specialty triage of biomedical abstracts or titles
• Proof-of-concept specialty routing of clinical free text (see limitations)
• Research on distant supervision and transfer learning in biomedical NLP

Not intended for: clinical decision support, diagnostic use, or any safety-critical
application.

Training and evaluation data

300,000 PubMed article titles from 77 medical journals, aggregated via the PubMed API.
Labels derived from the primary editorial scope of each source journal (distant supervision).

• IM sub-specialties (positive class): angiology, cardiology, gastroenterology,
  nephrology, hematology, pulmonology, endocrinology, rheumatology, geriatrics,
  intensive care medicine, infectiology
• Other disciplines (negative class): anesthesiology, otorhinolaryngology,
  gynecology, surgery, psychiatry, neurology

Dataset: Internal medicine and other specialties —
Kaggle

Training procedure

Fine-tuned from microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext:

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 640
• eval_batch_size: 1280
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 4

Training results

Evaluation results

Evaluated on a held-out test set of 100,000 titles (stratified, withheld from all training
steps):

In a prospective human study (n = 20 board-certified internists, 594 classifications),
the model achieved accuracy 0.931 vs. participant aggregate accuracy 0.773 (superiority test p < 10⁻¹⁴).

Label mapping

softmax(logits)[:, 1] gives P(Internal Medicine).

Limitations

• Trained on article titles only — short, structured academic text. Performance
  attenuates on clinical free text (discharge summaries, notes) due to domain shift.
• Labels derived from journal scope may not reflect clinical ground truth at disciplinary
  boundaries (e.g., intensive care / anesthesiology overlap).
• Evaluated on English-language text only.

Framework versions

• Transformers 4.31.0.dev0

• Pytorch 2.0.0

• Datasets 2.1.0

• Tokenizers 0.13.3

  Citation

  @misc{gamstaetter2023model,
    author       = {Gamstaetter, Thomas},
    title        = {im-bin-tf-abstr: Fine-tuned {PubMedBERT} for binary internal medicine          
  classification},                                                                                 
    year         = {2023},                                                                         
    howpublished = {Hugging Face},                                                                 
    url          = {https://huggingface.co/tgamstaetter/im-bin-tf-abstr}
  }                                                                                                
  

  Associated preregistration: OSF — DOI 10.17605/OSF.IO/XFDBV