longformer-gottbert-base-8192-aw512

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

first

This model is a fine-tuned version of longformer-gottbert-base-8192-aw512- on the a 500 million token subset of the german parts of the OSCAR dataset. It achieves the following results on the custom evaluation set:

• Loss: 1.4981

Model description

The weights of the model are initialized from the german version of Roberta gottbert-base. The local attention windows have a fixed size of 512 tokens across all layers. The maximum sequence length is 8192.

Intended uses & limitations

Longformer models enable processing long texts using a mixture of local attention on each subword token and task specific global attention on a subset of the tokens.

Training and evaluation data

The OSCAR dataset is freely avaible corpus of filtered web texts from the Common Crawl in various languages. We used the 2017 version of the dataset.

Training procedure

The model was trained with masked language modeling for 3 epochs on a customly created 500 million tokens subset of the german proportion of the OSCAR dataset. It was validated using 5% of the original subset.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 2
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 3.0
• mixed_precision_training: Native AMP

Training results

Framework versions

• Transformers 4.15.0
• Pytorch 1.10.1+cu113
• Datasets 1.17.0
• Tokenizers 0.10.3