PopEuroBERT-binary-610m

PopEuroBERT-610m

Binary Populism Classifier for German Bundestag Speeches

Table of Contents

1. Overview
2. Usage
3. Training Data
4. Training Procedure
5. Evaluation
6. Limitations
7. Ethical Considerations
8. License
9. Citation

Overview

This model is a fine-tuned version of EuroBERT-610m on the PopBERT dataset (sentence-level annotated German Bundestag speeches) for populist rhetoric classification. It predicts whether a given speech excerpt contains populist language.

Key Features:

• Trained on German Bundestag speeches sentence-level annotated for populism.
• Fine-tuned using 5-fold cross-validation.
• Optimized with decision threshold tuning.

Usage

To use the model in Python:

import torch
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification


model_id = "przvl/PopEuroBERT-binary-610m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(
  model_id, trust_remote_code=True
)

# define text to be predicted
text = (
    "Aber Ihnen fehlt eben der Mut, Ihnen fehlen die Visionen, um sich"
    "gegen die Konzerne und gegen die Lobbygruppen zur Wehr zu setzen."
)

inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# get classification probability
logits = outputs.logits
probs = torch.softmax(logits, dim=-1)  # shape [1, 2]
populist_prob = probs[0, 1].item()    # probability of class=1 (populist)

# use decision threshold 0.43
threshold = 0.43
label = "Populist" if populist_prob > threshold else "Neutral"
print(f"Predicted class: {label} (Confidence: {populist_prob:.2f})")

Predicted class: Populist (Confidence: 0.72)

Use decision threshold 0.43 for balanced performance.

Training Data

• Dataset: PopBERT
  • Sentence-level annotated German Bundestag speeches
  • train/test: 7017/1758
• Preprocessing:
  • Converted labels to binary format (populist = 1, neutral = 0).
  • Tokenized using EuroBERT tokenizer with a max length of 256 tokens.

Training Procedure

• Base Model: EuroBERT-610M
• Fine-tuning Approach:
  • Used Hugging Face Trainer for training.
  • Applied 5-fold cross-validation.
  • Decision threshold tuning on aggregated predictions.

Hyperparameters

• Mixed Precision (bf16): Used for efficiency on GPU.

Evaluation

For transparency, we compare this model with its smaller variant (PopEuroBERT-210m), both trained and evaluated on the same dataset and splits.

Test Set Performance (Threshold = 0.5)

Test Set Performance (Optimized Threshold)

PopEuroBERT-610m consistently outperforms the 210m variant across all metrics. It especially improves recall and F1 score, suggesting better identification of populist speech. The decision threshold (0.43) was tuned for balanced performance.

Limitations

• Domain Specificity: This model was trained on Bundestag speeches and may not generalize to all political discourse.
• Threshold Sensitivity: The decision threshold (0.43) was optimized for this dataset but may need adjustment for other corpora.
• Potential Bias: Political speech contains biases inherent in dataset labeling.

Ethical Considerations

• Not suitable for high-stakes decision-making. This model is meant for research purposes in political discourse analysis.
• Bias & Context Dependence: Populism is a complex concept. Automated detection should not replace human interpretation.
• Transparent Use: Users should document and validate model outputs in their research.

License

Released under the Apache 2.0 License.

Citation

If you use this model or its methodology, please cite:

• The original EuroBERT paper:

  @misc{boizard2025eurobertscalingmultilingualencoders,
        title={EuroBERT: Scaling Multilingual Encoders for European Languages},
        author={Nicolas Boizard and Hippolyte Gisserot-Boukhlef and Duarte M. Alves and André Martins and Ayoub Hammal and Caio Corro and Céline Hudelot and Emmanuel Malherbe and Etienne Malaboeuf and Fanny Jourdan and Gabriel Hautreux and João Alves and Kevin El-Haddad and Manuel Faysse and Maxime Peyrard and Nuno M. Guerreiro and Patrick Fernandes and Ricardo Rei and Pierre Colombo},
        year={2025},
        eprint={2503.05500},
        archivePrefix={arXiv},
        primaryClass={cs.CL},
        url={https://arxiv.org/abs/2503.05500}
  }
  

• The PopBERT dataset source:

  @article{Erhard_Hanke_Remer_Falenska_Heiberger_2025,
        title={PopBERT. Detecting Populism and Its Host Ideologies in the German Bundestag},
        volume={33},
        DOI={10.1017/pan.2024.12},
        number={1},
        journal={Political Analysis},
        author={Erhard, Lukas and Hanke, Sara and Remer, Uwe and Falenska, Agnieszka and Heiberger, Raphael Heiko},
        year={2025},
        pages={1–17}
  }