gbert-base-text_parts

Sequence classification model for text parts classification in German scientific texts

gbert-base-text_parts is a sequence classification model in the scientific domain in German, finetuned from the model gbert-base. It was trained using a custom annotated dataset of around 4,800 training and 1,400 test examples containing introduction, main part and conclusion text sequences from scientific papers and theses in german.

The two German BERT models — gbert-base and gbert-large — were evaluated on the same text classification task. This repository contains the model and results for gbert-base.

• gbert-base reached a maximum evaluation accuracy of 70.31%, with a minimum evaluation loss of 2.13.
• gbert-large achieved a higher peak accuracy of 78.50% and a lower evaluation loss of 1.67.

Despite its superior peak performance, gbert-large exhibited a consistent degradation in evaluation accuracy and an increase in loss over subsequent epochs. This pattern is characteristic of overfitting, where the model fits the training data too closely and fails to generalize well to unseen data. In contrast, gbert-base maintained more stable performance over time, making it a more robust choice in scenarios where computational resources or dataset size are limited.

Training

Training was conducted on a 10 epoch fine-tuning approach:

Training was conducted using a standard Text classification objective. The model achieves an accuracy of approximately 70% on the evaluation set.

Here are the overall final metrics on the test dataset after 10 epochs of training:

• Accuracy: 0.7030812324929971
• Loss: 2.7976272106170654