maude-clinicalbert

MAUDE Adverse Event Severity Classifier — Bio_ClinicalBERT (Phase 2)

Model Description

Fine-tuned Bio_ClinicalBERT classifier that assigns a severity label — Death (D) / Injury (I) / Malfunction (M) / Other (O) — to free-text adverse event narratives from the FDA's MAUDE database.

This is Phase 2 of a two-phase project. Phase 1 established a TF-IDF + Logistic Regression baseline; this model improves on that baseline specifically on the highest-stakes class (Death).

Training Data

• 154,776 MAUDE records (post-cleaning, UNKNOWN labels dropped)
• Source: openFDA MAUDE API
• Class distribution is imbalanced (Death prevalence ≈8%); handled via inverse-frequency class weights in the loss function

Training Procedure

Evaluation Results (5-fold CV)

Improvement over Phase 1 baseline: weighted F1 +0.0207, Death-class F1 +0.0617 (+6.2 pp) — the Death-class lift is the primary result this phase targeted.

Per-Fold Breakdown

Deployed Checkpoint

Fold 3 is the checkpoint pushed to this Hub repo, selected because it achieved the highest Death-class F1 (0.8418) across all five folds — not the highest weighted or macro F1 (fold 2 leads on both of those). This is a deliberate choice: in this application, a false negative on the Death class (a death narrative misclassified as Injury/Malfunction/Other) is the most clinically consequential error, so fold selection was optimized against that specific risk rather than an aggregate score.

Provenance note: this selection rationale was reconstructed and documented retroactively. At the time of the original push to this Hub repo, no model card or commit message recorded the selection criterion. Future pushes will include this documentation at push time, not after the fact.

Honest Limitations

• All folds peaked at the epoch-3 training cap — validation performance was still improving when training stopped due to patience=1. Extending training would likely add further gains; current numbers are conservative, not a ceiling.
• Other-class F1 (≈0.48) is the weakest class by a wide margin. This class is a small, semantically heterogeneous catch-all bucket rather than a coherent label, which limits how learnable it is regardless of further tuning. Treated as a known limitation, not an active optimization target for this phase.
• This is a research prototype. It is not validated for, and must not be used for, clinical decision-making.

Links

• GitHub repository — source code, CV splits, training scripts
• Phase 1 Space — TF-IDF baseline demo
• Phase 2 Space — this model, live demo

Author

Mukund Padmanabha — LinkedIn