Downloads · 30 days
37
9% of all-time downloads
lil-bevo
venkatasg/lil-bevo
lil-bevo is a fill-mask model from venkatasg. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the strict-small track.
All-time downloads
423
Public
Repo size
902 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin225 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- deberta-v2
- Access
- Public
- Created
- Jul 19, 2023
- Updated
- Aug 2, 2023
- SHA
- 7cc2d7ff
Try a prompt
Model card
From the Hugging Face model README
Lil-Bevo
Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the strict-small track.
TLDR:
-
Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
-
deberta-small-v3trained on mixture of MAESTRO and 10M tokens for 5 epochs. -
Model continues training for 50 epochs on 10M tokens with sequence length of 128.
-
Model is trained for 2 epochs with targeted linguistic masking with sequence length of 512.
This README will be updated with more details soon.
- Task
- Fill-Mask
- Library
- transformers
- Type
- deberta-v2
- License
- mit
- Languages
- en
- Created
- Jul 19, 2023
- Updated
- Aug 2, 2023