emotion-bert-base-binary-ensemble

emotion-bert-base-binary-ensemble

Nine independent bert-base-cased binary classifiers, one per Plutchik emotion plus Neutral, applied to English mobile app review sentences. This is the binary-ensemble formulation's best-performing configuration found in a broader comparison of encoder-only and decoder-only LLMs for this task: each emotion gets its own independent encoder and decision boundary rather than sharing one classification head, trained with focal loss on a training pool augmented with LLM-generated synthetic reviews (100 per emotion).

Under 10-fold cross-validation, this configuration reaches macro-F1 0.530 (+-0.074). It trails the multi-label formulation's best configuration (quim-motger/emotion-roberta-large-multilabel-genai-bce, macro-F1 0.591) on this corpus, though binary ensemble is markedly more robust to loss-reweighting miscalibration and does not share the multi-label head's complete failure on rare emotions at baseline.

This repo holds nine separate models, not one. Each subfolder is an independent bert-base-cased checkpoint with a single sigmoid output (num_labels=1): presence/absence of that one emotion. There is no shared backbone or parameter tying between them; assembling all nine into a prediction requires nine forward passes, one per emotion, matching how the paper reports inference cost for this formulation.

Labels / subfolders

Joy, Trust, Fear, Surprise, Sadness, Disgust, Anger, Anticipation, Neutral -- each is a subfolder in this repo containing its own independent binary classifier.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "quim-motger/emotion-bert-base-binary-ensemble"
emotions = ["Joy", "Trust", "Fear", "Surprise", "Sadness", "Disgust", "Anger", "Anticipation", "Neutral"]
tok = AutoTokenizer.from_pretrained(repo_id, subfolder=emotions[0])  # tokenizer is identical across subfolders

text = "This app used to be great but the last update broke everything."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)

probs = {}
for emotion in emotions:
    model = AutoModelForSequenceClassification.from_pretrained(repo_id, subfolder=emotion)
    with torch.no_grad():
        probs[emotion] = torch.sigmoid(model(**inputs).logits)[0, 0].item()

predicted = [e for e, p in probs.items() if p >= 0.5]
print(predicted, probs)

Note: the paper's decision rule assigns a label when p >= 0.5, falls back to the single most confident label if none cross that threshold, and caps predictions at 3 labels (the maximum cardinality observed in the ground truth). Replicate that rule yourself for scoring compatible with the paper.

Training data

Same recipe as the sibling multi-label model (quim-motger/emotion-roberta-large-multilabel-genai-bce): the human-labelled ground truth of 1,112 sentences (Motger et al., 2025), refit on the full training pool (all cross-validation folds combined) plus up to 100 LLM-generated synthetic reviews per emotion, but with focal loss (gamma=2.0) in place of BCE positive weighting, and one independent model per emotion instead of one shared head.

Citation

If you use this model, please cite the ground-truth dataset paper referenced in the parent replication package.