bert-base-uncased-danish

What is this?

A pre-trained BERT model (base version, ~110 M parameters) for Danish NLP. The model was not pre-trained from scratch but adapted from the English version with a tokenizer trained on Danish text.

How to use

Test the model using the pipeline from the 🤗 Transformers library:

from transformers import pipeline

pipe = pipeline("fill-mask", model="KennethTM/bert-base-uncased-danish")

pipe("Der var engang en [MASK]")

Or load it using the Auto* classes:

# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("KennethTM/bert-base-uncased-danish")
model = AutoModelForMaskedLM.from_pretrained("KennethTM/bert-base-uncased-danish")

Model training

The model is trained using multiple Danish datasets and a context length of 512 tokens.

The model weights are initialized from the English bert-base-uncased model with new word token embeddings created for Danish using WECHSEL.

Initially, only the word token embeddings are trained using 1.000.000 samples. Finally, the whole model is trained for 8 epochs.

Evaluation

The performance of the pretrained model was evaluated using ScandEval.