GloBERTise

This is an encoder Language Model pre-trained from scratch on transcriptions of the archives of the Dutch East India Company. It is therefore a model specialized on Early Modern Dutch as used in the archive (1602–1800). The model follows a RoBERTa architecture. It can be fine-tuned on any NLP task.

This version of the model is the best performing GloBERTise model when tested on binary event detection of the four I have pre-trained (in august 2025)

Comparison to other models: Adapted settings for 'num_training_steps' and 'num_warmup_steps' compared to GloBERTise-v01 and GloBERTise-v01-rerun, otherwise the same. Different seed compared to GloBERTise-rerun, same parameter settings.

See my GitHub repos

• for pre-training: https://github.com/globalise-huygens/GloBERTise
• for evaluation: https://github.com/globalise-huygens/GloBERTise-eval

And a small presentation: https://docs.google.com/presentation/d/1gkg5hChWAMXA6mxfgFkkvIieWdj_17yKitwBkBNcJBo/edit?usp=sharing

Most important parameter settings:

I pre-trained this model on National Supercomputer Snellius, which we were granted access to by the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-11854

I calculated a rough estimate of how much carbon was emitted by pre-training this model with https://mlco2.github.io/impact/#publish. Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.432 kg CO2/kWh. A cumulative of 24 hours of computation was performed on hardware of type A100 PCIe 40/80GB (TDP of 250W). Total emissions are estimated to be 2.59 kg CO2 of which 0 percents were directly offset.