DA-BERT_Old_News_V3

Model Card for Model ID

DA-Bert_Old_News_V3 is the third version of a transformer trained on Danish historical texts from the period during Danish Absolutism (1660-1849). It is developed by researchers at Aalborg University. The aim of the model is to create a domain-specific model to capture meaning from texts that are far enough removed in time that they no longer read like contemporary Danish.

<!-- Provide a quick summary of what the model is/does. -->

Fine-tuned DanskBERT model on MLM task. Training data: ENO (Enevældens Nyheder Online) – a corpus of news articles, announcements and advertisements from Danish and Norwegian newspapers from the period 1762 to 1848. The model was fine-tuned on version 1.0 of the ENO-dataset which consits of 4.9 million texts amouting to 474 million words. The data was created using a tailored Transkribus Pylaia-model and has an error rate of around 5% on word level.

Model Details

Architecture: DanskBERT

Fine-tuning Objective: Masked Language Modeling (MLM)

Sequence Length: 512 tokens

Tokenizer: Custom WordPiece tokenizer, trained on the ENO dataset

Developed by: CALDISS
Shared by: JohanHeinsen
Model type: BERT
Language(s) (NLP): Danish
License: MIT

Model Description

Third installment of the DA-Old-News models. This time trained on the full ENO-dataset. Still utilizes a custom tokenizer to enhance the models vocabulary coverage, OOV rates, and orthographic sensitivity. Consequently, the pretrained DanskBERT tokenizer was not employed, as its modern lexical inventory would have resulted in substantial token fragmentation and poor coverage of the archaic language inherent in the training data.

• Developed by: CALDISS, AAU
• Shared by: Johan Heinsen
• Model type: BERT-type architechture with custom tokenizer
• Language(s) (NLP): Danish
• License: MIT
• Finetuned from model: Vesteinn/DanskBERT

Model Sources

• Repository: https://github.com/CALDISS-AAU/OldNewsBERT
• Paper: In-progress

Uses

This model is designed for...

Domain-specific masked token prediction

Embedding extraction for semantic search

Further fine-tuning. More fine-tuning is needed to address specific use-cases.

The model is mostly intended for research purposes in the historical domain. Although not excluded to history.

The model can also serve as a baseline for further fine-tuning a historical BERT-based language model for either Danish or Scandinavian languages for textual or literary purposes.

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

Downstream Use

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Out-of-Scope Use

Bias, Risks, and Limitations

The model is heavily limited to the historical period the training data is from. Using this model for masked token prediction on modern Danish or even other scandinavian languages the performance of the model will vary. Further fine-tuning is therefore needed. Training data is from newspapers. A bias towards this type of material and therefore a particular manner of writing is inherent to the model. Newspapers are defined by highly literal language. The model's performance will therefore also vary if using it on more materials defined by figurative language. Small biases and risks also exists in the model based on the errors from the creation of the corpus. As mentioned there is an approximate 5% error on word level which continues onto the pre-trained model. Further work on addressing these biases and risks is planned further down the road.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model.

You can use this model directly with the transformers library:

from transformers import pipeline

model = "Da-BERT_Old_News_V3"
generator = pipeline("fill-mask", model=model)
generator(
    "I Tiisdags er hertil lykkelig hiemkommen den ene Grønlands Farer,som førtes af Commandeur Tagholm, samme har gjort en fordeelagtig [MASK],
    og skal have hiembragt henved 130 Qvardeeler Robbe=Spæk; Denne Fangst har de
    paa en Tid af 11 Dage gjort, fra den 14 til d. 25. April Syv andre fremmede Skibe
    som tillige med dem vare den indtrofne, have ogsaa gjort god Reyse."
)

Training Details

Training Data

Trained on the

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

[More Information Needed]

Training Hyperparameters

Training parameters: eval_strategy="steps", overwrite_output_dir=False, per_device_train_batch_size=16, # Training batch size gradient_accumulation_steps=4, # Accumulating the gradients before updating the weights per_device_eval_batch_size=32, # Evaluating batch size logging_steps=500, learning_rate=5e-5, save_steps=10000, max_steps=120000, save_total_limit=10, load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False, fp16=torch.cuda.is_available(), warmup_ratio=0.06, weight_decay=0.01, lr_scheduler_type="cosine", dataloader_num_workers=4, dataloader_pin_memory=True, save_on_each_node=False, ddp_find_unused_parameters=False, optim="adamw_torch",

• Training regime: fp16 mixed precision was used <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

[More Information Needed]

Results

[More Information Needed]

Summary

Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

[More Information Needed]

Software

[More Information Needed]

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Model Card Authors [optional]

SirMappel JohanHeinsen

Model Card Contact

• CALDISS
• Matias Appel ([email protected])
• Johan Heinsen ([email protected])