bert-base-greek-uncased-v6-finetuned-polylex-mg

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

bert-base-greek-uncased-v5-finetuned-polylex-mg

This model is a fine-tuned version of nlpaueb/bert-base-greek-uncased-v1 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.3369

Model description

In this work we appropriately adapt a corpus of multiword expressions in Modern Greek, namely PolylexMG, characterised by the features detailed above formulating its spectrum of idiosyncrasy to finetune Greek BERT transformer model for masked language modelling classification and tasks. The GREEK-BERT model is pre-trained on free text corpora extracted from (a) the Greek part of Wikipedia, (b) the Greek part of the European Parliament Proceedings Parallel Corpus (Europarl), and (c) the Greek part of OSCAR (Koutsikakis et al, 2020:113), this monolingual model is based on the architecture of BERT-BASE-UNCASED. Specifically, Greek BERT has been finetuned with expressions derived from each syntactic category as they are described in PolylexMG (Fotopoulou et al., 2023) that includes 6,000 Greek lexical entries dataset entailing frozen idioms which are semantically fixed with no paradigmatic variation (Lamiroy, 2003) and light verb constructions in which the semantics are traced in the predicative noun and not the verb (Anastassiadis-Symeonidis et al., 2020).

Results, intended uses & limitations

This subsection presents the experimental evaluation results for the MWE-fine-tuned Greek BERT model with respect to classification use case. The derived setup assumes that raw text in modern Greek that may contain multiple sentences is processed by the language model and reports class with regards to whether the text segment contains multiword stereotypical expressions or not. We compared the fine-tuned BERT model with a baseline logistic regression model. The latter is using as input the same word embeddings as the MWE-fine-tuned BERT model.
Greek-MWE-Bert was trained in a masked language model setting with full-expression-subdataset. The model perplexity was measured tο 303.21 before finetuning and 3.81 after finetuning demonstrating that the model has gained domain knowledge on multiword expressions. Qualitative outcomes presented in the following tables demonstrating the model performance in the case of 16 verbal constructs. The qualitative evaluation demonstrates that the fine-tuned model in all cases generates stereotypical multiword expressions while the original Greek BERT yields incomplete free-text related parts of sentences.

The finetuned model was further finetuned using for classification-oriented architecture with the classification-task-subdataset. The Bert classifier demonstrates an accuracy equivalent to 80% with a higher precision for free text reaching 80% and lower precision of 79% for MWE. In comparison the baseline classifier yields 70% for the free text and 67% for the MWE. We can observe that the two models have only 10% difference in accuracy despite the simplicity of baseline classifier. We can explain the small difference in the trained GreekBERT tokenizer that is used by both our model and the simplistic logistic regression model. However, the MWE-finetuned-Greek-BERT model can better capture sentences that contain MWEs due to the inherent benefits that the architecture offers.

Training and evaluation data

Sample of PolylexMG full expression subdataset

Sample of PolylexMG classification subdataset

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 512
• eval_batch_size: 512
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 500

Training results (Summary)

Framework versions

• Transformers 4.31.0
• Pytorch 2.0.1+cu118
• Datasets 2.14.3
• Tokenizers 0.13.3

Training results (Full)