Downloads · 30 days
123
4% of all-time downloads
megatron-bert-base-swedish-cased-600k
KBLab/megatron-bert-base-swedish-cased-600k
megatron-bert-base-swedish-cased-600k is a fill-mask model from KBLab. Use it when you need the model to fill a missing word. It is set up for transformers.
This BERT model was trained using the Megatron-LM library. The size of the model is a regular BERT-base with 110M parameters. The model was trained on about 70GB of data, consisting mostly of OSCAR and Swedish newspap…
All-time downloads
3.1K
Public
Parameters
135M
1.1 GB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin542 MB · 50%
Parameter types
How the weights are stored.
F32135M · 100%
Model card
From the Hugging Face model README
Megatron-BERT-base Swedish 600k
This BERT model was trained using the Megatron-LM library. The size of the model is a regular BERT-base with 110M parameters. The model was trained on about 70GB of data, consisting mostly of OSCAR and Swedish newspaper text curated by the National Library of Sweden.
Training was done for 600k training steps. Its sister model used the same setup, but was instead trained for only 125k steps.
The model has three sister models trained on the same dataset:
Acknowledgements
We gratefully acknowledge the HPC RIVR consortium (https://www.hpc-rivr.si) and EuroHPC JU (https://eurohpc-ju.europa.eu) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (https://www.izum.si).
- Task
- Fill-Mask
- Library
- transformers
- Type
- megatron-bert
- Languages
- sv
- Created
- Mar 2, 2022
- Updated
- Mar 21, 2023