Downloads · 30 days
21
6% of all-time downloads
lil-bevo-x
venkatasg/lil-bevo-x
lil-bevo-x is a fill-mask model from venkatasg. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the strict track.
All-time downloads
377
Public
Repo size
1.8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin447 MB · 95%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- deberta-v2
- Access
- Public
- Created
- Jul 19, 2023
- Updated
- Aug 25, 2023
- SHA
- b2361ea8
Try a prompt
Model card
From the Hugging Face model README
Lil-Bevo-X
Lil-Bevo-X is UT Austin's submission to the BabyLM challenge, specifically the strict track.
Model training regime:
- 5 epochs on MAESTRO dataset (85M non-language music tokens) combined with strict small dataset.
- 50 epochs of pretraining with sequence length of 128 on strict dataset.
- 150 epochs of pretraining with sequence length of 512 on strict dataset.
- 10 epochs of targeted MLM.
This README will be updated with more details soon.
- Task
- Fill-Mask
- Library
- transformers
- Type
- deberta-v2
- License
- mit
- Languages
- en
- Created
- Jul 19, 2023
- Updated
- Aug 25, 2023