biomedbert-ohsumed23

BiomedBERT — OHSUMED-23

23-way classifier for MEDLINE abstracts (MeSH disease chapters C01–C23), fine-tuned from microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext on AxelNoun/ohsumed-23-single-label (revision c2e11d41da4d537366e75e02bc3b21a29a4978e0).

This is not a set of “23 cardiovascular categories”. GNN papers often phrase it that way; that is incorrect. The 23 labels are MeSH chapters C01–C23. C14 (Cardiovascular Diseases) is one of those 23 classes. Spot-check: duodenitis → C06 (digestive).

Results (test, 4,043 documents)

Decomposition: DistilBERT → bert-base = +2.0 acc / +5.3 macro-F1 (capacity); bert-base → BiomedBERT = +8.8 / +14.9 (domain).

Macro-F1 excl. C22 is the unweighted mean of the other 22 class F1s (0.699 × 23/22).

Uncertainty

Two distinct dispersions are reported; they answer different questions:

• Initialization variability — seeds 42 / 43 / 44, same split: accuracy 0.775 ± 0.003 · macro-F1 0.699 ± 0.005 · top-3 0.934 ± 0.002. This speaks to training stability.
• Sampling variability — 95% bootstrap on the 4,043 test documents, 1,000 resamples: macro-F1 in [0.680, 0.715]. This is the one to use when comparing to a figure published elsewhere.

Released weights are seed 42 (accuracy 0.774, macro-F1 0.699). The table above reports the three-seed mean; a gap on the order of 0.001 between the two is expected.

Positioning on the TextGCN protocol

Reading. Only the “paired” row is directly comparable to the cited work: it uses the same preprocessing. On that footing, BiomedBERT is slightly ahead in accuracy (+2.6) and slightly behind in macro-F1 (−1.7): the two approaches are at the same level. We claim no superiority.

The released model consumes raw text and posts higher absolute scores, but that number is not comparable to anything in the literature — it only shows what you get without degrading the input.

What TextGCN preprocessing costs a domain encoder

Absent from the cited literature: same BiomedBERT, same seed 42, same 3,357 / 4,043 split; only the input changes (preprocess_textgcn.py, NLTK stopwords + tokens with frequency < 5, vocabulary computed on train alone). From 0.774 / 0.699 (raw) to 0.741 / 0.662 (filtered): −3.3 accuracy points, −3.7 macro-F1.

Mechanism. The freq < 5 filter on ~3,357 training documents shrinks the kept vocabulary to 8,290 tokens (after stopword drop). Those are precisely the rare technical terms — where a WordPiece trained on PubMed beats a generalist vocabulary. The TextGCN protocol therefore systematically handicaps subword transformers relative to graph methods, which rely on frequent-word co-occurrences and lose less from that filtering. Confirming signature: macro-F1 drops more than accuracy (−3.7 vs −3.3), consistent with rare terms carrying mostly rare classes.

Scope of the comparison

Three models were compared, chosen to isolate domain effect at fixed architecture. This protocol shows that pretraining domain drives most of the gain — not that BiomedBERT is the best possible choice. Not evaluated: BioLinkBERT, recent long-context encoders, and a zero-shot LLM baseline that would set the floor without training. Contributions welcome.

Dominant confusions

Row-normalized matrix: evidence/confusion_matrix.png (also under outputs/biomedbert-ohsumed23/ locally).

Top pairs: C23 → C14 (50) · C23 → C10 (42) · C23 → C21 / C02 → C20 (24 each).

C23 asymmetry. Strongly outbound flow: 264 C23→X vs 107 X→C23 (ratio 2.47). Precision 0.592, recall 0.370, F1 0.455 (n=419). The asymmetry makes the mechanism visible: the model is not symmetrically confusing C23 with something else — it under-emits C23. When it predicts C23 it is right six times out of ten; it only catches one third of true C23. Signature of a catch-all class learned to fire only when no organ signal is present. Second-largest macro-F1 hole after C22.

Headline figure: macro-F1 0.699 (23 classes). Diagnostic on the 21 well-formed classes — structural criterion fixed a priori, not “the two worst scores”: C22 has no usable support, C23 is not a domain but a residual chapter — : 0.744. A weak class for any other reason would stay in the average.

Dominant confusions are not domain-to-domain ambiguity but the residual nature of chapter C23, which absorbs signs and symptoms independently of the organ system. A C23-labeled abstract talks about a sign; its text is saturated with organ vocabulary; the model predicts the organ. That is a labeling granularity problem, not MeSH polyhierarchy.

Only one top pair matches a documented MeSH cross-listing: C02→C20. 23/24 cases mention HIV / AIDS / immunodeficiency / HTLV (1987–1991 AIDS literature). C04↔C08 and C10↔C20 remain rare in the matrix (4 and 1).

Limitations

C22 is never predicted

C22 - Animal Diseases scores F1 0.000 on all three seeds. With roughly a dozen test examples and similar training support, the model never learned to emit it: no input will receive this label. Any application that needs C22 detection must use another mechanism.

Metric consequence: a dead class costs 1/23 mechanically, i.e. 4.3 macro-F1 points. Hence dual reporting — 0.699 on 23 classes, 0.731 on the 22 classes with usable support.

Scores are not calibrated

Weighted cross-entropy (α = 0.5) used against class imbalance shifts calibration. Displayed scores are relative confidence scores, not probabilities. Temperature scaling on validation would be needed before any threshold-based use.

Intrinsic label ambiguity

Two distinct mechanisms coexist; they are not fixed the same way.

1. Granularity (C23). An abstract about a general clinical sign (hemorrhage, shock, pain, syncope…) will be routed to an organ chapter. Predictable behavior: C23 is a MeSH catch-all, not a domain; the model under-emits it (precision 0.592, recall 0.370; 264 outbound vs 107 inbound leaks). Demo users should know this.

2. Polyhierarchy. MeSH is not a tree. Illustration (app.py examples):

• Lung Neoplasms: C04 and C08. The model predicts C08 (0.90) — legitimate; the corpus mostly labels “lung carcinoma / cancer” as C08.
• Multiple Sclerosis: C10 and C20. The model predicts C20 (0.86); the corpus says C20 (train 10/12, test 15/15).

The top-1 → top-3 gap (0.774 → 0.933) mostly captures these two effects — hence Top-3 in the demo.

Future work (formulation, not capacity)

The principled fix is not a better encoder but a better task formulation: multi-label or hierarchical labeling that does not force a choice between the sign (C23) and the organ, or between two cross-listed chapters. That is the conclusion the matrix supports.

Temporal drift

Abstracts from 1987–1991. Dated terminology, drugs, and protocols. No guarantee of generalization to contemporary biomedical literature.

Not a medical device

Bibliographic categorization only. No diagnostic or clinical decision-support use.

License and terms of use

Two layers apply jointly:

1. Base model — microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext is distributed under the MIT license.
2. Training data — the OHSUMED corpus is redistributed under its own terms, stricter than MIT. Fine-tuned weights inherit those usage constraints.

How far a data license propagates into fine-tuned weights is not settled law. If commercial use is contemplated, check the source OHSUMED collection terms before any deployment. This card is not legal advice.

Reproduction

python train.py --dataset_id AxelNoun/ohsumed-23-single-label --dataset_revision c2e11d41da4d537366e75e02bc3b21a29a4978e0 --train_batch_size 8 --gradient_accumulation_steps 2 --eval_batch_size 16 --max_length 448 --seed 42
python eval_checkpoint.py --model_id outputs/biomedbert-ohsumed23/final
python analyze_results.py --run_dir outputs/biomedbert-ohsumed23

Demo: $env:MODEL_ID="AxelNoun/biomedbert-ohsumed23"; python app.py

Citation

@inproceedings{yao2019graph,
  title={Graph Convolutional Networks for Text Classification},
  author={Yao, Liang and Mao, Chengsheng and Luo, Yuan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={33},
  number={01},
  pages={7370--7377},
  year={2019}
}

@article{lv2024rbgat,
  title={{RB-GAT}: A Text Classification Model Based on {RoBERTa}-{BiGRU} with Graph {ATtention} Network},
  author={Lv, Shanshan and Dong, Jinming and Wang, Chao and Wang, Xianwei and Bao, Zhiyu},
  journal={Sensors},
  volume={24},
  number={11},
  pages={3365},
  year={2024}
}