gbert-large-ft-edu-redux

German BERT large fine-tuned to predict educational requirements

This is a fine-tuned version of the German BERT large language model deepset/gbert-large. The multilabel task this model was trained on was to predict education requirements from job ad texts. The dataset used for training is not available to the public. The 7 labels in the task are (in the classification head order):

• 'Bachelor'
• 'Berufsausbildung'
• 'Doktorat oder äquivalent'
• 'Höhere Berufsausbildung'
• 'Master'
• 'Sonstiges'
• 'keine Ausbildungserfordernisse'

The number of representatives of these labels in each of the splits (train/test/val) of the dataset is summarized in the following table:

Performance

Training consisted of minimizing the binary cross-entropy (BCE) loss between the model's predictions and the actual labels in the training set. During training, a weighted version of the label ranking average precision (LRAP) was tracked for the testing set. LRAP measures what fraction of higher-ranked labels produced by the model were true labels. To account for the label imbalance, the rankings were weighted so that improperly ranked rare labels are penalized more than their more frequent counterparts. After training was complete, the model with highest weighted LRAP was saved.

LRAP: 0.96

See also:

• deepset/gbert-base
• deepset/gbert-large
• gonzpen/gbert-base-ft-edu-redux

Authors

Rodrigo C. G. Pena: rodrigocgp [at] gmail.com