Downloads · 30 days
4.1K
2% of all-time downloads
sloberta
EMBEDDIA/sloberta
sloberta is a fill-mask model from EMBEDDIA. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as cc-by-sa-4.0.
All-time downloads
230K
Public
Repo size
886 MB
Likes
5
Public
Repo makeup
Click a slice to open those files.
.bin443 MB · 99%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- cc-by-sa-4.0
- Model type
- camembert
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Nov 24, 2021
- SHA
- 49db152a
Model card
From the Hugging Face model README
Usage
Load in transformers library with:
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/sloberta")
model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/sloberta")
SloBERTa
SloBERTa model is a monolingual Slovene BERT-like model. It is closely related to French Camembert model https://camembert-model.fr/. The corpora used for training the model have 3.47 billion tokens in total. The subword vocabulary contains 32,000 tokens. The scripts and programs used for data preparation and training the model are available on https://github.com/clarinsi/Slovene-BERT-Tool
SloBERTa was trained for 200,000 iterations or about 98 epochs.
Corpora
The following corpora were used for training the model:
- Gigafida 2.0
- Kas 1.0
- Janes 1.0 (only Janes-news, Janes-forum, Janes-blog, Janes-wiki subcorpora)
- Slovenian parliamentary corpus siParl 2.0
- slWaC
- Task
- Fill-Mask
- Library
- transformers
- Type
- camembert
- License
- cc-by-sa-4.0
- Languages
- sl
- Hf Inference
- error
- Created
- Mar 2, 2022
- Updated
- Nov 24, 2021