MeMo-BERT-01

MeMo-BERT-01

MeMo-BERT-01 is a pre-trained language model for historical Danish and Norwegian literary texts (1870–1900).
It was introduced in Al-Laith et al. (2024) as part of the first dedicated PLMs for historical Danish and Norwegian.

Model Description

• Architecture: BERT-base (12 layers, hidden size 768, 12 attention heads, vocab size 30k)
• Pre-training strategy: Trained from scratch on the MeMo corpus (no prior pre-training)
• Training objective: Masked Language Modeling (MLM, 15% masking)
• Training data: MeMo Corpus v1.1 (839 novels, ~53M words, 1870–1900)
• Hardware: 2 × A100 GPUs
• Training time: ~44 hours

This model represents the baseline historical-domain model trained entirely on 19th-century Scandinavian novels.

Intended Use

• Primary tasks:

  • Sentiment Analysis (positive, neutral, negative)
  • Word Sense Disambiguation (historical vs. modern senses of skæbne, "fate")

• Intended users:

  • Researchers in Digital Humanities, Computational Linguistics, and Scandinavian Studies.
  • Historians of literature studying 19th-century Scandinavian novels.

• Not intended for:

  • Contemporary Danish/Norwegian NLP tasks.
  • High-stakes applications (e.g., legal, medical, political decision-making).

Training Data

• Corpus: MeMo Corpus v1.1 (Bjerring-Hansen et al. 2022)
• Time period: 1870–1900
• Size: 839 novels, 690 MB, 3.2M sentences, 52.7M words
• Preprocessing: OCR-corrected, normalized to modern Danish spelling, tokenized, lemmatized, annotated

Evaluation

Benchmarks

Comparison

MeMo-BERT-01 performs worse than MeMo-BERT-03 (continued pre-training), highlighting the limitations of training from scratch on historical data without leveraging contemporary PLMs.

Limitations

• Trained only from scratch on ~53M words (relatively small for BERT training).
• Underperforms compared to continued pre-training (MeMo-BERT-03).
• Domain-specific to late 19th-century novels.
• OCR and normalization errors may remain in training corpus.

Ethical Considerations

• All texts are public domain (authors deceased).
• Datasets released under CC BY 4.0.
• No sensitive personal data involved.

Citation

If you use this model, please cite:

@inproceedings{al-laith-etal-2024-development,
    title = "Development and Evaluation of Pre-trained Language Models for Historical {D}anish and {N}orwegian Literary Texts",
    author = "Al-Laith, Ali and Conroy, Alexander and Bjerring-Hansen, Jens and Hershcovich, Daniel",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    pages = "4811--4819",
    url = "https://aclanthology.org/2024.lrec-main.431/"
}