Downloads · 30 days
36
7% of all-time downloads
deberta-wiki-2006
mikesong724/deberta-wiki-2006
deberta-wiki-2006 is a fill-mask model from mikesong724. Use it when you need the model to fill a missing word. It is set up for transformers.
All-time downloads
504
Public
Repo size
1.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin499 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- deberta
- Access
- Public
- Created
- Jun 25, 2022
- Updated
- Jun 25, 2022
- SHA
- 97086580
Try a prompt
Model card
From the Hugging Face model README
DeBERTa trained from scratch
Source data: https://dumps.wikimedia.org/archive/2006/
Tools used: https://github.com/mikesong724/Point-in-Time-Language-Model
2006 wiki archive 2.7 GB trained 24 epochs = 65GB
GLUE benchmark
cola (3e): matthews corr: 0.2848
sst2 (3e): acc: 0.8876
mrpc (5e): F1: 0.8033, acc: 0.7108
stsb (3e): pearson: 0.7542, spearman: 0.7536
qqp (3e): acc: 0.8852, F1: 0.8461
mnli (3e): acc_mm: 0.7822
qnli (3e): acc: 0.8715
rte (3e): acc: 0.5235
wnli (5e): acc: 0.3099
- Task
- Fill-Mask
- Library
- transformers
- Type
- deberta
- Created
- Jun 25, 2022
- Updated
- Jun 25, 2022