pubmedul2_small

PubMedUL2 & MedUL2

Model Description

PubMedUL2 and MedUL2 are a family of domain-specific UL2/T5-style encoder–decoder language models pretrained on large-scale biomedical and medical corpora using the UL2 (Mixture-of-Denoisers) objective.

• PubMedUL2 models are pretrained on 25 million PubMed abstracts
• MedUL2 models are pretrained on PubMed abstracts + clinical notes + additional medical documents
• All models use a T5-efficient architecture, inspired by Google’s efficient T5 variants

These checkpoints are pretraining-only models and must be fine-tuned before use on downstream tasks.

---

Pretraining Objective: UL2 (Mixture-of-Denoisers)

These models were pretrained using UL2, a unified framework that formulates language modeling objectives as denoising tasks.

UL2 introduces a Mixture-of-Denoisers (MoD) approach that samples from multiple denoising paradigms during pretraining.

Denoising Tasks

UL2 pretraining uses a mixture of three denoising tasks:

1. R-denoising (Regular Span Corruption)

   • Equivalent to standard T5 span corruption
   • Optimized for language understanding tasks

2. X-denoising (Extreme Span Corruption)

   • Uses very large masked spans
   • Encourages long-form generation and abstraction

3. S-denoising (Sequential / PrefixLM)

   • Prefix language modeling similar to causal LM
   • Suitable for sequence-to-sequence and generative tasks

Paradigm Tokens (Mode Switching)

During pretraining, a paradigm token is inserted at the beginning of each input:

Important:
For best performance, the same token should be prepended during fine-tuning and inference.

---

Architecture

• Encoder–decoder Transformer (T5-style)
• Uses T5-efficient architecture
• Compatible with Hugging Face T5ForConditionalGeneration

---

Intended Uses

These models are intended to be fine-tuned for:

• Biomedical and clinical text classification
• Question answering
• Summarization of medical literature or clinical notes
• Text generation in medical contexts

---

Limitations

• ❌ Not instruction-tuned
• ❌ No supervised training
• ❌ Not suitable for zero-shot use

These checkpoints are self-supervised pretraining models only and require task-specific fine-tuning.

---

Fine-Tuning Recommendations

• Avoid mixed precision (fp16 / bf16) initially
  • Fine-tuning is more stable in fp32
• Always prepend one of [NLU], [NLG], or [S2S] to input text
• Suggested defaults:
  • Classification / QA → [NLU]
  • Causal or generative tasks → [S2S]
  • Mixed tasks → [NLG]

---

Model Parameter Summary

---

Named Entity Recognition (NER) Evaluation

We evaluate PubMedUL2 and MedUL2 models on a biomedical Named Entity Recognition (NER) task using multiple matching criteria to better capture boundary-level performance.

The evaluation reports entity-level F1 scores across different biomedical entity types and model sizes.

Exact Match F1

An entity prediction is considered correct only if both the entity span and label exactly match the gold annotation.

---

Partial Match F1

A prediction is counted as correct if it partially overlaps with a gold entity of the same type.

---

IoU Match F1

Predictions are evaluated using Intersection-over-Union (IoU) overlap between predicted and gold spans, providing a softer boundary-based metric.

---

Observations

• MedUL2 models generally outperform PubMedUL2 on clinical-heavy entity types such as disease and chemical
• Performance improves consistently from tiny → base models
• Boundary-sensitive metrics (Partial / IoU) show significantly higher scores than Exact Match, highlighting boundary ambiguity in biomedical NER

---

Acknowledgements

This project would not have been possible without compute generously provided by Google TPU Research Cloud.

Thanks to:

• The Finnish-NLP authors for releasing the UL2 objective code, task definitions, and guidance
• Yeb Havinga for help getting started with the t5x framework

---

License

Please refer to the individual model repositories for license and access details, which may vary depending on training data sources.