Downloads · 30 days
121
1% of all-time downloads
bert-large-swedish-cased
AI-Nordics/bert-large-swedish-cased
bert-large-swedish-cased is a fill-mask model from AI-Nordics. Use it when you need the model to fill a missing word. It is set up for transformers.
This model follows the Bert Large model architecture as implemented in Megatron-LM framework. It was trained with a batch size of 512 in 600k steps. The model contains following parameters: <figure
All-time downloads
10.3K
Public
Repo size
2.7 GB
Likes
11
Public
Repo makeup
Click a slice to open those files.
.bin1.3 GB · 100%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- megatron-bert
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Feb 15, 2022
- SHA
- b7925d4c
Model card
From the Hugging Face model README
A Swedish Bert model
Model description
This model follows the Bert Large model architecture as implemented in Megatron-LM framework. It was trained with a batch size of 512 in 600k steps. The model contains following parameters:
<figure>| Hyperparameter | Value |
|---|---|
| \(n_{parameters}\) | 340M |
| \(n_{layers}\) | 24 |
| \(n_{heads}\) | 16 |
| \(n_{ctx}\) | 1024 |
| \(n_{vocab}\) | 30592 |
Training data
The model is pretrained on a Swedish text corpus of around 85 GB from a variety of sources as shown below.
<figure>| Dataset | Genre | Size(GB) |
|---|---|---|
| Anföranden | Politics | 0.9 |
| DCEP | Politics | 0.6 |
| DGT | Politics | 0.7 |
| Fass | Medical | 0.6 |
| Författningar | Legal | 0.1 |
| Web data | Misc | 45.0 |
| JRC | Legal | 0.4 |
| Litteraturbanken | Books | 0.3O |
| SCAR | Misc | 28.0 |
| SOU | Politics | 5.3 |
| Subtitles | Drama | 1.3 |
| Wikipedia | Facts | 1.8 |
Intended uses & limitations
The raw model can be used for the usual tasks of masked language modeling or next sentence prediction. It is also often fine-tuned on a downstream task to improve its performance in a specific domain/task. <br> <br>
How to use
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("AI-Nordics/bert-large-swedish-cased")
model = AutoModelForMaskedLM.from_pretrained("AI-Nordics/bert-large-swedish-cased")
- Task
- Fill-Mask
- Library
- transformers
- Type
- megatron-bert
- Languages
- sv
- Created
- Mar 2, 2022
- Updated
- Feb 15, 2022