DA-BERT_Old_News_V1

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

DA-Bert_Old_News_V1 is the first version of a transformer trained on Danish historical texts from the period during Danish Absolutism (1660-1849). It is created by researchers at Aalborg University. The aim of the model is to create a domain-specific model to capture meaning from texts that are far enough removed in time that they no longer read like contemporary Danish.

Model Details

Pretrained BERT model on MLM task. Training data: ENO (Enevældens Nyheder Online) – a corpus of news articles, announcements and advertisements from Danish and Norwegian newspapers from the period 1762 to 1848. The model has been trained on a subset consisting of about 260m words. The data was created using a tailored Transkribus Pylaia-model and has an error rate of around 5% on word level.

Model Description

Architecture: BERT

Pretraining Objective: Masked Language Modeling (MLM)

Sequence Length: 512 tokens

Tokenizer: Custom WordPiece tokenizer

<!-- Provide a longer summary of what this model is. -->

• Developed by: CALDISS
• Shared by: JohanHeinsen
• Model type: BERT
• Language(s) (NLP): Danish
• License: MIT

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/CALDISS-AAU/OldNewsBERT
• Paper [optional]: In-progress

Uses

This model is designed for...

• Domain-specific masked token prediction

• Embedding extraction for semantic search

• Further fine-tuning

• Further fine-tuning is needed for adressing specific use-cases.

• Further plans for retraining on more data and annotated data for fine-tuning is in the works.

• The model is mostly intended for research purposes in the historical domain. Although not excluded to history.

• The model can also serve as a baseline for further fine-tuning a historical BERT-based language model for either Danish or Scandinavian languages for textual or literary purposes.

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

• This model can be used out-of-the-box for domain-specific masked token prediction.
• The model can also be used for basic mean-pooled embeddings on similar data. Results on this may vary as this model is only trained on the MLM task using the transformer trainer-framework.

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

As the model is trained on the ENO dataset the model is not used for modern Danish text because of its inherent historical training data.

Bias, Risks, and Limitations

The model is heavily limited to the historical period the training data is from. Using this model for masked token prediction on modern Danish or even other scandinavian languages the performance of the model will vary. Further fine-tuning is therefore needed. Training data is from newspapers. A bias towards this type of material and therefore a particular manner of writing is inherent to the model. Newspapers are defined by highly literal language. The model's performance will therefore also vary if using it on more materials defined by figurative language. Small biases and risks also exists in the model based on the errors from the creation of the corpus. As mentioned there is an approximate 5% error on word level which continues onto the pre-trained model. Further work on addressing these biases and risks is planned further down the road.

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

The model is based on historical texts that express a range of antiquated worldviews. These include racist, anti-democratic and patriarchal sentiments. This makes it utterly unfit for many use cases. It can, however, be used to examine such biases in Danish history.

How to Get Started with the Model

Use the code below to get started with the model.

Training Details

The model is trained using the Hugging Face trainer API.

• Model trained for roughly 45 hours on the provided HPC-system.
• The MLM-prob was defined as .15

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

[More Information Needed]

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing

Texts shorter than 35 chars were removed. Texts including a predetermined amount of German, Latin or rare words were removed. Extra whitespaces were also removed.

Training Hyperparameters

• Training regime: [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision --> Training arguments: eval_strategy="steps", overwrite_output_dir=True, num_train_epochs=15, per_device_train_batch_size=16, gradient_accumulation_steps=4, per_device_eval_batch_size=64, logging_steps=500, learning_rate=5e-5, save_steps=1000, save_total_limit=5, load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, fp16=torch.cuda.is_available(), warmup_steps=2000, warmup_ratio=0.03, weight_decay=0.01, lr_scheduler_type="cosine", dataloader_num_workers=4, dataloader_pin_memory=True, save_on_each_node=False, ddp_find_unused_parameters=False, optim="adamw_torch", local_rank=local_rank,

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

Cross-entropy loss. Standard use for BERT with MLM training.

Avg. Loss on test-set

Perplexity. Calculated based on loss value.

Results

Loss: 2.08

Avg. Loss on test-set: 2.07

Perplexity: 7.65

Summary

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

Technical Specifications

Model Architecture and Objective

Compute Infrastructure

Ucloud-cloud infrastructure available at the Danish universities

Hardware

Hardware Type: 64 (Intel Xeon Gold 6326), 256 GB memory, 4 Nividia A10 Hours used: 44 hours 34 minutes Cloud Provider: Ucloud SDU Compute Region: Cloud services based at University of Southern Denmark, Aarhus University and Aalborg University

Software

Python 3.12.8

Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Model Card Authors

• Matias Appel ([email protected])
• Johan Heinsen ([email protected])

Model Card Contact

CALDISS, AAU: www.caldiss.aau.dk