roberta

Model trained from roberta-base on the go_emotions dataset for multi-label classification.

go_emotions is based on Reddit data and has 28 labels. It is a multi-label dataset where one or multiple labels may apply for any given input text, hence this model is a multi-label classification model with 28 'probability' float outputs for any given input text. Typically a threshold of 0.5 is applied to the probabilities for the prediction for each label.

The model was trained using AutoModelForSequenceClassification.from_pretrained with problem_type="multi_label_classification" for 3 epochs with a learning rate of 2e-5 and weight decay of 0.01.

Evaluation (of the 28 dim output via a threshold of 0.5 to binarize each) using the dataset test split gives:

• Micro F1 0.585
• ROC AUC 0.751
• Accuracy 0.474

But the metrics would be more meaningful when measured per label given the multi-label nature.

Additionally some labels (E.g. gratitude) when considered independently perform very strongly with F1 around 0.9, whilst others (E.g. relief) perform very poorly. This is a challenging dataset. Labels such as relief do have much fewer examples in the training data (less than 100 out of the 40k+), but there is also some ambiguity and/or labelling errors visible in the training data of go_emotions that is suspected to constrain the performance.