emotion-roberta-large-multilabel-genai-bce

emotion-roberta-large-multilabel-genai-bce

Fine-tuned roberta-large multi-label classifier over Plutchik's eight emotions plus Neutral, applied to English mobile app review sentences.

This is the best-performing encoder-only configuration found in a broader comparison of encoder-only and decoder-only LLMs for this task: a shared multi-label classification head trained with a BCE loss that up-weights each emotion's positive class in proportion to its rarity, on a training pool augmented with LLM-generated synthetic reviews (100 per emotion) to fix the baseline model's complete failure to detect Fear, Surprise and Anger.

Under 10-fold cross-validation on the human-labelled ground truth, this configuration reaches macro-F1 0.591 (+-0.054), recovering all three previously undetected emotions (Anger 0->0.501, Fear 0->0.410, Surprise 0->0.346 F1) with no regression on the four majority emotions.

This specific checkpoint is refit on the full training pool (all cross-validation folds combined, plus the same synthetic augmentation), so it has no internal held-out set of its own -- treat the cross-validated macro-F1 above as the expected-performance estimate for this recipe, not a number to reproduce exactly from this single checkpoint.

Labels

Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, Anticipation, Neutral -- multi-label (0 or more may apply per sentence).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("quim-motger/emotion-roberta-large-multilabel-genai-bce")
model = AutoModelForSequenceClassification.from_pretrained("quim-motger/emotion-roberta-large-multilabel-genai-bce")

emotions = ["Joy", "Trust", "Fear", "Surprise", "Sadness", "Disgust", "Anger", "Anticipation", "Neutral"]

text = "This app used to be great but the last update broke everything."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

predicted = [e for e, p in zip(emotions, probs) if p >= 0.5]
print(predicted, probs.tolist())

Note: the paper's decision rule assigns a label when p >= 0.5, falls back to the single most confident label if none cross that threshold, and caps predictions at 3 labels (the maximum cardinality observed in the ground truth). Replicate that rule yourself for scoring compatible with the paper.

Training data

• Human-labelled ground truth: 1,112 sentences from 257 apps across 10 Google Play categories, adapted from Plutchik's eight basic emotions plus Neutral (Motger et al., 2025).
• Synthetic augmentation: up to 100 additional LLM-generated reviews per emotion (claude-opus-4-6, few-shot with guideline examples plus dataset exemplars), selected via an embedding-based augmentation-utility ranking before injection.

Citation

If you use this model, please cite the ground-truth dataset paper referenced in the parent replication package.