Downloads · 30 days
33
0% of all-time downloads
megatron-bert-large-swedish-cased-110k
KBLab/megatron-bert-large-swedish-cased-110k
megatron-bert-large-swedish-cased-110k is a machine learning model from KBLab. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
This BERT model was trained using the Megatron-LM library. The size of the model is a regular BERT-large with 340M parameters. The model was trained on about 70GB of data, consisting mostly of OSCAR and Swedish newspa…
All-time downloads
52.5K
Public
Repo size
1.5 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin741 MB · 100%
At a glance
- Library
- transformers
- Model type
- megatron-bert
- Access
- Public
- Created
- Mar 17, 2022
- Updated
- May 3, 2022
- SHA
- 1599fbd1
Model card
From the Hugging Face model README
Megatron-BERT-large Swedish 110k
This BERT model was trained using the Megatron-LM library. The size of the model is a regular BERT-large with 340M parameters. The model was trained on about 70GB of data, consisting mostly of OSCAR and Swedish newspaper text curated by the National Library of Sweden.
Training was done for 110k training steps using a batch size of 8k; the number of training steps is set to 500k, meaning that this version is a checkpoint. The hyperparameters for training followed the setting for RoBERTa.
The model has three sister models trained on the same dataset:
and a later checkpoint
Acknowledgements
We gratefully acknowledge the HPC RIVR consortium (https://www.hpc-rivr.si) and EuroHPC JU (https://eurohpc-ju.europa.eu) for funding this research by providing computing resources of the HPC system Vega at the Institute of Information Science (https://www.izum.si).
- Library
- transformers
- Type
- megatron-bert
- Languages
- sv
- Created
- Mar 17, 2022
- Updated
- May 3, 2022