Downloads · 30 days
9
1% of all-time downloads
roberta-small-bulgarian
iarfmoose/roberta-small-bulgarian
roberta-small-bulgarian is a fill-mask model from iarfmoose. Use it when you need the model to fill a missing word. It is set up for transformers.
The RoBERTa model was originally introduced in this paper. This is a smaller version of RoBERTa-base-bulgarian with only 6 hidden layers, but similar performance.
All-time downloads
1.6K
Public
Repo size
1.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.h5498 MB · 42%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- roberta
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- May 20, 2021
- SHA
- 1a9eed8d
Model card
From the Hugging Face model README
RoBERTa-small-bulgarian
The RoBERTa model was originally introduced in this paper. This is a smaller version of RoBERTa-base-bulgarian with only 6 hidden layers, but similar performance.
Intended uses
This model can be used for cloze tasks (masked language modeling) or finetuned on other tasks in Bulgarian.
Limitations and bias
The training data is unfiltered text from the internet and may contain all sorts of biases.
Training data
This model was trained on the following data:
- bg_dedup from OSCAR
- Newscrawl 1 million sentences 2017 from Leipzig Corpora Collection
- Wikipedia 1 million sentences 2016 from Leipzig Corpora Collection
Training procedure
The model was pretrained using a masked language-modeling objective with dynamic masking as described here
It was trained for 160k steps. The batch size was limited to 8 due to GPU memory limitations.
- Task
- Fill-Mask
- Library
- transformers
- Type
- roberta
- Languages
- bg
- Created
- Mar 2, 2022
- Updated
- May 20, 2021