Downloads · 30 days
9
2% of all-time downloads
deberta-wiki-2010
mikesong724/deberta-wiki-2010
deberta-wiki-2010 is a fill-mask model from mikesong724. Use it when you need the model to fill a missing word. It is set up for transformers.
All-time downloads
448
Public
Repo size
998 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin499 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- deberta
- Access
- Public
- Created
- Jul 7, 2022
- Updated
- Jul 7, 2022
- SHA
- 2e299bea
Try a prompt
Model card
From the Hugging Face model README
DeBERTa trained from scratch
continued training from https://huggingface.co/mikesong724/deberta-wiki-2006
Source data: https://dumps.wikimedia.org/archive/2010/
Tools used: https://github.com/mikesong724/Point-in-Time-Language-Model
2010 wiki archive 6.1 GB trained 18 epochs = 108GB + 2006 (65GB)
GLUE benchmark
cola (3e): matthews corr: 0.3640
sst2 (3e): acc: 0.9106
mrpc (5e): F1: 0.8505, acc: 0.7794
stsb (3e): pearson: 0.8339, spearman: 0.8312
qqp (3e): acc: 0.8965, F1: 0.8604
mnli (3e): acc_mm: 0.8023
qnli (3e): acc: 0.8889
rte (3e): acc: 0.5271
wnli (5e): acc: 0.3380
- Task
- Fill-Mask
- Library
- transformers
- Type
- deberta
- Created
- Jul 7, 2022
- Updated
- Jul 7, 2022