bio-electra-mid-1_2m

Bio-ELECTRA Mid 1.2m (cased)

Pretrained (from scratch for 1.2 million steps) mid-sized (50 million parameters) ELECTRA discriminator model on 2021 Base PubMed abstracts and PMC open access papers with a domain specific word piece vocabulary generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. This model is case-sensitive: it makes a difference between english and English.

Intended uses & limitations

This model is mostly intended to be fine-tuned on a downstream biomedical domain task.

Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence to make decisions, such as classification, information retrieval, relation extraction or question answering.

Training data

The pretraining corpus was built using 21.2 million PubMed abstracts from the January 2021 baseline distribution. To build the corpus, title and abstract text sentences were extracted resulting in a corpus of 3.6 billion words. The PMC open access corpus (January 2021) is a 12.3 billion words corpus built using the sentences extracted from the sections of PMC open access papers excluding the references sections.

Training procedure

The training procedure follows the original ELECTRA training.

Preprocessing

A domain specific vocabulary of size 31,620 is generated using SentencePiece byte-pair-encoding (BPE) model from PubMed abstract texts. The title and abstract text sentences were extracted using an in-house sentence segmenter trained on biomedical text. The sentences are pre-tokenized using an in-house biomedical tokenizer for proper tokenization of biomedical entities such as gene/protein names, organisms, antibodies, cell lines. The SentencePiece BPE vocabulary of word pieces are applied during pre-training to the properly tokenized and segmented sentences. For the PMC open access corpus, JATS XML files for the full text papers are parsed to extract sections excluding the reference section and section title and section body is processed in the same fashion as the PubMed abstracts corpus.

Pretraining

The model is pretrained on a single 8 core version 3 tensor processing unit (TPU) with 128 GB of RAM for 1,200,000 steps with a batch size of 256. The first 1,000,000 steps are pre-trained on PubMed abstracts. After that, the model is pre-trained for another 200,000 steps on PMC open access papers. The training parameters were the same as the original ELECTRA base model. The model has 50M parameters, 12 transformers layers with hidden layer size of 512 and 8 attention heads.

BibTeX entry and citation info

 @inproceedings{ozyurt-etal-2021-detecting,
    title = "Detecting Anatomical and Functional Connectivity Relations in Biomedical Literature via Language Representation Models",
    author = "Ozyurt, Ibrahim Burak  and
      Menke, Joseph  and
      Bandrowski, Anita  and
      Martone, Maryann",
    editor = "Beltagy, Iz  and
      Cohan, Arman  and
      Feigenblat, Guy  and
      Freitag, Dayne  and
      Ghosal, Tirthankar  and
      Hall, Keith  and
      Herrmannova, Drahomira  and
      Knoth, Petr  and
      Lo, Kyle  and
      Mayr, Philipp  and
      Patton, Robert M.  and
      Shmueli-Scheuer, Michal  and
      de Waard, Anita  and
      Wang, Kuansan  and
      Wang, Lucy Lu",
    booktitle = "Proceedings of the Second Workshop on Scholarly Document Processing",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.sdp-1.4",
    doi = "10.18653/v1/2021.sdp-1.4",
    pages = "27--35",
    abstract = "Understanding of nerve-organ interactions is crucial to facilitate the development of effective bioelectronic treatments. Towards the end of developing a systematized and computable wiring diagram of the autonomic nervous system (ANS), we introduce a curated ANS connectivity corpus together with several neural language representation model based connectivity relation extraction systems. We also show that active learning guided curation for labeled corpus expansion significantly outperforms randomly selecting connectivity relation candidates minimizing curation effort. Our final relation extraction system achieves $F_1$ = 72.8{\%} on anatomical connectivity and $F_1$ = 74.6{\%} on functional connectivity relation extraction.",
}