illiberal-speech-cd-scorer

Illiberal Speech — Concentration–Dispersion (CD) Scorer

A fully fine-tuned DeBERTa-v3-base regression model that scores political-speech paragraphs on the concentration–dispersion (CD) dimension of political discourse, on a 0–10 scale:

• 0 — very liberal/democratic: endorses democratic principles such as checks and balances, transparency, free media, and civil society.
• 5 — neutral: mentions norms or institutions without a clear stance.
• 10 — very illiberal/authoritarian: attacks democratic norms and accountability, advocates fewer checks on power, expresses clear support for authoritarian uses of power.

This is the second stage of the two-stage Illiberal Speech Indicators (ISI) pipeline: paragraphs are first filtered for topical relevance by the companion classifier, illiberal-speech-cd-relevance, and relevant paragraphs are scored by this model.

The models were developed for:

Maerz, Seraphine F., Dean G. Schafer, and Carsten Q. Schneider. Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline. (2026)

where the ISI pipeline scores 28,227 speeches by 453 political leaders in 123 countries (1989–2024), yielding over half a million semantically coherent paragraphs.

The ISI model family

Model details

• Architecture: microsoft/deberta-v3-base encoder (full fine-tune, 183.8M trainable parameters), mean pooling over the last hidden state, and a linear regression head. Max sequence length 512.
• Training data: 2,495 paragraphs (2,020 train / 225 validation / 250 held-out) sampled from the speech corpus and scored 0–10 by GPT-4o under a detailed codebook, with human-adjudicated gold-standard subsets used for validation (see the paper's Supplementary Materials).
• Training: MSE loss on standardized labels, lr 2e-5, effective batch size 16, early stopping on validation loss (best epoch 13), seed 1999, PyTorch 2.9.1+cu130 on an NVIDIA RTX PRO 4000 Blackwell (24 GB).

Performance

On the 250-paragraph held-out set (hand-checked by the authors): R² = 0.782, MAE = 0.78, RMSE = 1.00, Pearson r = 0.89 (on the 0–10 scale).

In the paper's cross-validation against OpenAI's gpt-5.2 on human-coded gold-standard data, the DeBERTa scorers match or slightly outperform gpt-5.2 across both ISI dimensions, with lower error and bias (paper, Supplementary Materials II §8.3).

⚠️ The model and scaler are a matched pair

The model predicts in standardized units (labels were z-scored with a StandardScaler fit on the full training data: mean = 4.467695390781563, scale = 2.1093737018905045). Predictions must be inverse-transformed with the scaler shipped in this repo — illiberal-speech-cd-scorer-scaler.pkl — to land on the 0–10 scale. Using any other scaler will silently shift and stretch every score.

Usage

import joblib
import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import DebertaV2Tokenizer, DebertaV2Model

REPO = "d-schafer/illiberal-speech-cd-scorer"

class DebertaForRegression(nn.Module):
    def __init__(self):
        super().__init__()
        self.deberta = DebertaV2Model.from_pretrained("microsoft/deberta-v3-base")
        self.regressor = nn.Linear(self.deberta.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        h = self.deberta(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        m = attention_mask.unsqueeze(-1).expand(h.size()).float()
        pooled = (h * m).sum(1) / m.sum(1).clamp(min=1e-9)
        return self.regressor(pooled)

model_path  = hf_hub_download(REPO, "illiberal-speech-cd-scorer.pth")
scaler_path = hf_hub_download(REPO, "illiberal-speech-cd-scorer-scaler.pkl")

model = DebertaForRegression()
model.load_state_dict(torch.load(model_path, map_location="cpu", weights_only=True))
model.eval()
scaler = joblib.load(scaler_path)

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")

texts = ["An independent judiciary and a free press are the backbone of our democracy."]
enc = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    pred = model(enc["input_ids"], enc["attention_mask"])
scores = scaler.inverse_transform(pred.numpy())  # 0-10 scale; higher = more concentrating/illiberal
print(scores.ravel().tolist())

Requires torch, transformers, sentencepiece, scikit-learn, and joblib. The scaler was pickled with scikit-learn 1.8.0; older versions may emit a version warning on load.

Files

Intended use & limitations

Designed for paragraph-length segments (~200 words) of political leaders' speeches in English; non-English speeches in the underlying corpus were translated to English before processing. Intended to be run on paragraphs pre-filtered by the CD relevance classifier — scores for off-topic text are not meaningful. Paragraph-level scores are noisy by nature; the paper aggregates them to speaker-year estimates with a latent trajectory model before analysis. Not validated for other text domains, other languages, or sentence-/document-length inputs.

Citation

If you use this model, please cite the paper:

@unpublished{maerz2026listening,
  title  = {Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline},
  author = {Maerz, Seraphine F. and Schafer, Dean G. and Schneider, Carsten Q.},
  year   = {2026},
  note   = {Working paper}
}

An interactive dashboard for exploring the data is available at https://validation.shinyapps.io/speeches/.