Downloads · 30 days
34
3% of all-time downloads
clinical-pubmed-bert-base-128
Tsubasaz/clinical-pubmed-bert-base-128
clinical-pubmed-bert-base-128 is a fill-mask model from Tsubasaz. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
A BERT model pre-trained on PubMed abstracts, and continual pre-trained on clinical notes (MIMIC-III). We try combining two domains that have fewer overlaps with general knowledge text corpora: EHRs and biomedical pap…
All-time downloads
1.1K
Public
Repo size
876 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin438 MB · 100%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- bert
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Jan 27, 2022
- SHA
- 73a95af2
Try a prompt
Datasets
Model card
From the Hugging Face model README
ClinicalPubMedBERT
Description
A BERT model pre-trained on PubMed abstracts, and continual pre-trained on clinical notes (MIMIC-III). We try combining two domains that have fewer overlaps with general knowledge text corpora: EHRs and biomedical papers. We hope this model can serve better results on clinical-related downstream tasks such as readmissions.
This model is trained on 500000 clinical notes randomly sampled from MIMIC datasets, with 120k steps of training. We also used whole word masking to enhance the coherence of the language model. All notes are chunked into a length of 128 tokens.
Pre-trained model: https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
- Task
- Fill-Mask
- Library
- transformers
- Type
- bert
- License
- mit
- Languages
- en
- Created
- Mar 2, 2022
- Updated
- Jan 27, 2022