Downloads · 30 days
48
0% of all-time downloads
keptlongformer
whaleloops/keptlongformer
keptlongformer is a fill-mask model from whaleloops. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as apache-2.0.
<span style="font-size:larger;"KEPTlongfomer</span is a medical knowledge enhanced version of Longformer that was further pre-trained using contrastive learning. The model achieves SOTA performance on auto ICD coding…
All-time downloads
12.4K
Public
Repo size
1.3 GB
Likes
3
Public
Repo makeup
Click a slice to open those files.
.bin633 MB · 100%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- apache-2.0
- Model type
- longformer
- Access
- Public
- Created
- Sep 9, 2022
- Updated
- Nov 13, 2022
- SHA
- 6805f213
Try a prompt
Model card
From the Hugging Face model README
<span style="font-size:larger;">KEPTlongfomer</span> is a medical knowledge enhanced version of Longformer that was further pre-trained using contrastive learning. The model achieves SOTA performance on auto ICD coding on MIMIC-III as of 11/12/2022. A sister model for better performance is available here.
Pre-training
We initialized this model from clinical longformer.
And then pretrained with Hierarchical Self-Alignment Pretrain (HSAP) using Knowledge Graph UMLS. This includes (a) Hierarchy, (b) Synonym, (c) Abbreviation. For more info, see section 3.3 in paper. The learning rate was 5e-5, weight decay was 0.01, adam epsilon was 1e-5.
Usage
See our github for how to use this with prompts on auto ICD coding.
With the following result:
| Metric | Score |
|---|---|
| rec_micro | =0.5729403619819988 |
| rec_macro | =0.11342156911120573 |
| rec_at_8 | =0.4094837705486378 |
| rec_at_75 | =0.8470734920535119 |
| rec_at_50 | =0.8005338782352 |
| rec_at_5 | =0.2891628170355805 |
| rec_at_15 | =0.5768778119750537 |
| prec_micro | =0.6411968713105065 |
| prec_macro | =0.12227610414493029 |
| prec_at_8 | =0.7760972716488731 |
| prec_at_75 | =0.197504942665085 |
| prec_at_50 | =0.2768090154211151 |
| prec_at_5 | =0.8483392645314354 |
| prec_at_15 | =0.6178529062870699 |
| f1_micro | =0.6051499904242899 |
| f1_macro | =0.11768251595637802 |
| f1_at_8 | =0.536107150495997 |
| f1_at_75 | =0.32032290907137506 |
| f1_at_50 | =0.411373195944102 |
| f1_at_5 | =0.43131028155283435 |
| f1_at_15 | =0.5966627077602488 |
| auc_micro | =0.9651754312635265 |
| auc_macro | =0.8566590059725866 |
| acc_micro | =0.43384592341105344 |
| acc_macro | =0.08639139221100567 |
- Task
- Fill-Mask
- Library
- transformers
- Type
- longformer
- License
- apache-2.0
- Languages
- en
- Created
- Sep 9, 2022
- Updated
- Nov 13, 2022