illiberal-speech-ei-relevance

Illiberal Speech — Exclusive–Inclusive (EI) Relevance Filter

A DeBERTa-v3-base + LoRA binary classifier that flags whether a political-speech paragraph is topically relevant to the exclusive–inclusive (EI) dimension of political discourse: cultural identity, group boundaries, minority rights, pluralism, and tolerance.

This is the first stage of the two-stage Illiberal Speech Indicators (ISI) pipeline. Paragraphs flagged as relevant are then scored 0–10 by the companion regression model, illiberal-speech-ei-scorer.

The models were developed for:

Maerz, Seraphine F., Dean G. Schafer, and Carsten Q. Schneider. Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline. (2026)

where the ISI pipeline scores 28,227 speeches by 453 political leaders in 123 countries (1989–2024), yielding over half a million semantically coherent paragraphs.

The ISI model family

Model details

• Architecture: microsoft/deberta-v3-base encoder with LoRA adapters (r=16, α=32, dropout=0.1, target modules: query_proj, key_proj, value_proj), mean pooling over the last hidden state, and a linear 2-class classification head. Max sequence length 512.
• Training data: 42,953 training paragraphs (3,817 validation / 955 test) drawn from a stratified random sample of the speech corpus. Binary relevance labels were derived from GPT-4.1 relevance ratings (0–100 scale; ≥65 → relevant, ≤35 → not relevant, ambiguous middle excluded).
• Training: early stopping on validation F1 (best epoch 5, val F1 = 0.943), seed 1969, PyTorch 2.9.1+cu130 on an NVIDIA RTX PRO 4000 Blackwell (24 GB).

Performance

On the held-out test split (n = 955): F1 (positive class) = 0.897, F1 macro = 0.929, AUC = 0.986.

On an independent hand-coded holdout (n = 299 paragraphs labelled by the authors), at the recommended decision threshold of 0.20: recall = 0.878, precision = 0.800, F1 (positive) = 0.837, AUC = 0.857. The threshold was chosen by grid search on this holdout to favor recall: false positives are tolerable because the downstream scorer sees them anyway, while false negatives are lost from the index entirely.

Usage

The .pth file contains the full model state dict (LoRA weights and the classification head). The lora_adapters/ folder holds the PEFT adapters for the encoder only — it does not include the classification head, so use the .pth for working predictions:

import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import DebertaV2Tokenizer, DebertaV2Model
from peft import LoraConfig, get_peft_model

REPO = "d-schafer/illiberal-speech-ei-relevance"
THRESHOLD = 0.20  # recommended decision threshold

class DebertaForClassification(nn.Module):
    def __init__(self):
        super().__init__()
        self.deberta = DebertaV2Model.from_pretrained("microsoft/deberta-v3-base")
        hidden_size = self.deberta.config.hidden_size
        lora_config = LoraConfig(
            r=16, lora_alpha=32, lora_dropout=0.1,
            target_modules=["query_proj", "key_proj", "value_proj"],
            bias="none", task_type="FEATURE_EXTRACTION",
        )
        self.deberta = get_peft_model(self.deberta, lora_config)
        self.classifier = nn.Linear(hidden_size, 2)
        self.dropout = nn.Dropout(0.1)

    def forward(self, input_ids, attention_mask):
        h = self.deberta(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        m = attention_mask.unsqueeze(-1).expand(h.size()).float()
        pooled = (h * m).sum(1) / m.sum(1).clamp(min=1e-9)
        return self.classifier(self.dropout(pooled))

model_path = hf_hub_download(REPO, "illiberal-speech-ei-relevance.pth")
model = DebertaForClassification()
model.load_state_dict(torch.load(model_path, map_location="cpu", weights_only=True))
model.eval()

tokenizer = DebertaV2Tokenizer.from_pretrained(REPO, subfolder="tokenizer")

texts = ["We must defend our nation against those who would replace our culture."]
enc = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]), dim=1)[:, 1]
relevant = probs >= THRESHOLD
print(probs.tolist(), relevant.tolist())

Requires torch, transformers, peft, and sentencepiece.

Files

Intended use & limitations

Designed for paragraph-length segments (~200 words) of political leaders' speeches in English; non-English speeches in the underlying corpus were translated to English before processing. Performance is not validated for other text domains (social media, news, parliamentary debate), other languages, or sentence-/document-length inputs. The recommended threshold (0.20) favors recall by design; raise it if your application needs higher precision.

Citation

If you use this model, please cite the paper:

@unpublished{maerz2026listening,
  title  = {Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline},
  author = {Maerz, Seraphine F. and Schafer, Dean G. and Schneider, Carsten Q.},
  year   = {2026},
  note   = {Working paper}
}

An interactive dashboard for exploring the data is available at https://validation.shinyapps.io/speeches/.