english_politic_RoBERTa-large

english_politic_RoBERTa-large

This classifier predicts whether an English text is political (politic) or not (other). It is FacebookAI/roberta-large fine-tuned on SinclairSchneider/trainset_political_text_yes_no_english, a set of about 45k news and blog articles from TIME, Military Times and Security Affairs. Each article is labeled by the section or outlet it was published in.

The training script is in this repository: train.py.

The model reaches 97.9% accuracy / 0.979 weighted F1 on a held-out split of its own training data. On an outside dataset (BBC News) it ranks texts well (ROC-AUC 0.97), but the default 0.5 threshold produces many false positives, especially for sport and business news. For out-of-domain text, use a stricter threshold such as 0.9–0.98. See Evaluation.

The German counterpart is SinclairSchneider/german_politic_DeBERTa-v2-base.

Related paper: Ideology Prediction of German Political Texts (Schneider, Steuber, Schneider & Dreo Rodosek, ICWSM 2026). This model was developed in the same research project. The paper uses the German counterpart as a pre-filter that removes non-political texts before political ideology is predicted.

[!IMPORTANT] Before using the model, note:

• Pass truncation=True. Inputs longer than 512 tokens otherwise crash the pipeline (RuntimeError: index 514 is out of bounds…, or a CUDA device-side assert on GPU).
• Replace non-breaking spaces (\xa0) with regular spaces. In the training data they appeared only in one non-political source, so the model learned them as a cue for other. Three of them inserted into a political article flip 25% of political BBC articles to other.
• Don't rely on the 0.5 threshold outside the news domain the model was trained on. On BBC News it labels 79% of sport and 65% of business articles as politic.
• Short texts get low-confidence scores. Headlines and single sentences often score between 0.5 and 0.75, e.g. "Trump and Putin agree on limited ceasefire" → politic 0.65.

Quick start

from transformers import pipeline

political_classifier = pipeline("text-classification", model="SinclairSchneider/english_politic_RoBERTa-large")

political_classifier("The Senate passed the bill by a vote of 52 to 48, sending the immigration reform package to the president's desk.")
# [{'label': 'politic', 'score': 0.9991005659103394}]

political_classifier("The new smartphone features a larger battery, a faster processor and an improved camera system.")
# [{'label': 'other', 'score': 0.9969991445541382}]

political_classifier("Trump and Putin agree on limited ceasefire")
# [{'label': 'politic', 'score': 0.6543024778366089}]

political_classifier("Franck Ribéry and Arjen Robben celebrate their comeback at FC Bayern at the \"Beckenbauer Cup\". They look as good as in their best days and awaken a great longing at the record champions.")
# [{'label': 'other', 'score': 0.7409901022911072}]

The last two examples are the English translations of the examples in the German model's card. The outputs shown were produced with transformers 5.15 / torch 2.13. Scores may differ in the last decimal places with other versions or hardware.

Recommended usage for real data (long texts, batching, custom threshold)

import re
from transformers import pipeline

political_classifier = pipeline("text-classification", model="SinclairSchneider/english_politic_RoBERTa-large")

def normalize(text):
    # Non-breaking spaces and typographic quotes push the model towards "other" (see "Shortcut probes")
    text = (text.replace("\xa0", " ")
                .replace("“", '"').replace("”", '"')
                .replace("‘", "'").replace("’", "'"))
    return re.sub(r"\s+", " ", text).strip()

texts = [
    "Lawmakers are debating a bill to raise the minimum wage.",
    "This recipe needs two cups of flour and a pinch of salt.",
    " ".join(["The parliament debated the new budget proposal for several hours."] * 120),  # > 512 tokens
]

results = political_classifier([normalize(t) for t in texts], truncation=True, top_k=None, batch_size=32)
p_politic = [{d["label"]: d["score"] for d in r}["politic"] for r in results]
print([round(p, 4) for p in p_politic])
# [0.9129, 0.01, 0.9984]

THRESHOLD = 0.95  # stricter than 0.5; see "Choosing a threshold"
print([p >= THRESHOLD for p in p_politic])
# [False, False, True]

With top_k=None, the pipeline returns the scores of both labels. Note that the minimum-wage sentence is a political text that falls below the strict threshold: single sentences rarely reach 0.95. Choose the threshold according to the length of your inputs and how you weigh false positives against false negatives.

Without the pipeline

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "SinclairSchneider/english_politic_RoBERTa-large"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

texts = ["The prime minister announced a snap election for next month.",
         "The band announced a world tour starting in March."]
enc = tokenizer(texts, truncation=True, max_length=512, padding=True, return_tensors="pt")
with torch.no_grad():
    probs = model(**enc).logits.softmax(dim=-1)

for text, p in zip(texts, probs):
    print(model.config.id2label[int(p.argmax())], round(float(p[1]), 4), text)
# politic 0.991 The prime minister announced a snap election for next month.
# other 0.0013 The band announced a world tour starting in March.

Model details

Uses

Direct use

• Flagging or filtering political news articles in large English text collections, e.g. as a pre-filter before manual coding or downstream political-text analysis.
• Ranking documents by how likely they are to be political. The ROC-AUC is high even out of domain, so the scores are more reliable than the hard 0.5 decisions.

Downstream use

• As a starting checkpoint for fine-tuning on a target domain, e.g. parliamentary speeches, social media or a different political definition. A few hundred in-domain labels are likely to fix most of the domain-shift problems described below.

Out-of-scope use

• Political stance, ideology, party or sentiment detection. The model only tells whether a text is political, not what position it takes.
• Detecting misinformation, propaganda or harmful content.
• Automated decisions about people, e.g. moderation or flagging of individuals' posts, without human review.
• Non-English text. For German, use the German model.
• Short texts (tweets, headlines, chat messages) or sports content without validating the threshold on your own data first.

Training data

The model was trained on SinclairSchneider/trainset_political_text_yes_no_english. Labels come from the section or outlet; no article was labeled by hand.

The articles date mostly from the 2010s and come mostly from US outlets. The other class contains no sport, travel, food, science or religion content. This explains the sport false positives in the out-of-domain evaluation. The dataset card describes the composition, statistics and known artifacts in detail.

Training procedure

Preprocessing

As implemented in train.py:

1. Only the article body (content) was used. The title was not used.
2. A keyword filter removed source fingerprints: the strings Paganini Security, Security Affair, Pierluigi, Paganini, opting Newsletter, Newsletter, breaking news, email opting, directly inbox, email and inbox were deleted from every text.
3. Empty texts were dropped (178 rows), and so were duplicate (label, text) pairs (184 rows). That left 45,108 texts (23,340 other / 21,768 politic).
4. The texts were shuffled and split randomly 80/20 with Dataset.train_test_split(0.2), giving 36,086 train / 9,022 eval texts. No seed was set for the shuffle or the split, so the exact split can't be reproduced.
5. Texts were tokenized with the RoBERTa BPE tokenizer and truncated to 512 tokens. Batches were padded dynamically (DataCollatorWithPadding).

The keyword filter removes words but not the remaining sentence fragments or character-level formatting. Some source artifacts therefore stayed in the training text, and the model picked them up (see Shortcut probes).

Training hyperparameters

The published model.safetensors stores the epoch-3 weights as float32 (1.42 GB). The values are identical to the bf16 checkpoint, just upcast.

Speeds, sizes, times

Evaluation

1. In-distribution: held-out 20% of the training data

Scores on 9,022 held-out texts, from the same sources and the same period as the training data. Precision, recall and F1 are weighted over both classes.

The split is random, so source-specific artifacts appear in both train and eval data. Read these numbers as an upper bound for news articles similar to the training data.

2. Out-of-domain: BBC News

Setup. The original, cased BBC News dataset (Greene & Cunningham, 2006) contains 2,225 BBC articles from 2004–2005 in five sections. After removing exact duplicates, 2,127 remain: politics 403, business 503, entertainment 369, sport 505, tech 347.

Target: politics → politic, all other sections → other. This test differs from the training data in outlet (BBC), country (UK), period (2004–2005) and topic mix: sport never appeared in training. The model input was title + "\n\n" + body, truncated to 512 tokens. Four input variants were evaluated: the full article, the first sentence of the body only, the headline only, and the full article in lowercase.

Results at the default threshold (0.5)

Confusion matrix, full article at 0.5: TN 933 · FP 791 · FN 1 · TP 402.

Share of each BBC section predicted as politic

Choosing a threshold

Full articles, BBC News:

These thresholds were picked on the same data they are scored on, so the numbers are optimistic. Validate a threshold on a sample of your own data.

What goes wrong

• Sport is the main weakness. The other class in training had no sport, and sports reporting shares vocabulary with world news: nations, teams, coaches, "campaign", "defeat". Sport articles get a median politic score of 0.71. Most of them are only moderately confident, though, so a 0.9–0.95 threshold removes almost all of them. Examples: "Campbell to be Lions consultant" 0.995, "Bomb threat at Bernabeu stadium" 0.983, "Iranian misses Israel match" 0.981, "Spain coach faces racism inquiry" 0.968.
• Business false positives are often political economy. Examples: "Iraqi voters turn to economic issues" 0.999, "Algeria hit by further gas riots" 0.999, "Minister hits out at Yukos sale" 0.999, "Saudi ministry to employ women" 0.999. Many of these would count as political under most definitions, so part of this "error" comes from the BBC section labels.
• The same applies to entertainment and tech. Examples: "Uganda bans Vagina Monologues" 0.985, "Iran jails blogger for 14 years" 0.995, "Satellite mapping aids Darfur relief" 0.997, "Ink helps drive democracy in Asia" 0.998.
• Recall is almost perfect. Only 1 of 403 political articles was missed: "UK firms 'embracing e-commerce'" (0.40).
• Out-of-domain predictions are overconfident. 62% of predictions have a confidence ≥ 0.9, but only 80% of those are correct.
• Case doesn't matter much. Lowercased input performs as well as cased input (AUC 0.979 vs. 0.971).

3. Short sentences

These 16 hand-written sentences were not drawn from any dataset. The table shows P(politic):

At 0.5, 14 of 16 are correct. Both errors are topics that were missing from the training data (sport, travel), and the scores for everyday sentences sit close to 0.5.

4. Shortcut probes

To test whether the model learned source formatting instead of topic, typical training-source artifacts were inserted into BBC articles and the predictions compared.

Military Times newsletter text → pushes towards politic. The newsletter fragment left in the Military Times training texts after keyword filtering ("Fear no longer. Be the first to hear about , as it happens. … Sign up for the Early Bird Brief,") was inserted after the first paragraph of the 1,724 non-political BBC articles. A neutral newsletter sentence of similar length served as a control:

Security Affairs formatting → pushes towards other. These were applied to the bodies of the 403 political BBC articles:

The non-breaking space is the strongest single cue. In the training data it occurred only in Security Affairs (other). Web-scraped text often contains non-breaking spaces, so normalize whitespace before classifying (see the normalize() function above).

Bias, risks, and limitations

• The definition of "political" is set by the source sections. It includes all military and defense reporting, as well as TIME World, which also covers disasters, crime and celebrity news abroad. It excludes IT-security reporting, even on state hacking and election interference. TIME US and Ideas were left out of training entirely.
• US-centric, 2010s news. Politicians, parties and issues from that time and place are over-represented. Performance on other countries' politics, other periods, or other genres (speeches, manifestos, social media, forums) is untested. It's probably lower.
• Missing negative topics. Sport, travel, food, science and religion were never shown as other, so the model scores them unpredictably, often as politic.
• Source shortcuts. The model reacts to formatting artifacts (non-breaking spaces, newsletter boilerplate) that have nothing to do with politics. Adversarial or accidental formatting can flip predictions.
• Overconfidence out of domain. High scores outside the training domain are not guarantees. Use a stricter threshold and spot-check.
• Truncation. Only the first 512 tokens (≈ 380 words) are read. About half of the training articles were longer, and a long text that turns political late may be missed.
• Copyright of the training data. The model was trained on copyrighted news articles whose dataset license is unknown. Check whether this matters for your use case.

Recommendations

1. Always call the pipeline with truncation=True, and normalize whitespace and quotes before inference.
2. Choose the threshold on a small labeled sample of your own data. For news-like text, a threshold of 0.9–0.98 gave a much better precision/recall balance than 0.5 on BBC News.
3. Use the probability as a ranking score and review borderline cases (roughly 0.5–0.95) by hand when precision matters.
4. Fine-tune further on in-domain data, especially if your data contains sport or other topics missing from training, or consists of short texts.

Environmental impact

• Hardware: 4 × NVIDIA RTX 6000 Ada Generation (48 GB each), PyTorch DataParallel, on-premise university GPU server (no cloud provider).
• Training time: ≈ 0.37 h for the published run.
• Energy: at most ≈ 0.44 kWh for the GPUs (4 × 300 W TDP × 0.37 h).
• Carbon emitted: ≈ 0.2 kg CO₂eq, assuming ~0.4 kg CO₂eq/kWh. This is an estimate in the style of Lacoste et al. (2019).

These figures cover only the published run. Development runs (one earlier complete run of ≈ 17 min and several short debugging runs) are not included.

Technical specifications

Model architecture and objective

• RoBERTa-large encoder: 24 layers, hidden size 1,024, 16 attention heads, feed-forward size 4,096, GELU, absolute position embeddings (514 positions → 512 usable tokens), byte-level BPE vocabulary of 50,265 tokens. The model is case-sensitive.
• Classification head (RobertaClassificationHead): dropout → dense 1,024→1,024 + tanh → dropout → linear 1,024→2 applied to the <s> token.
• Objective: softmax cross-entropy over the two labels (problem_type: single_label_classification).

Software

• Training: Python 3.10.12, PyTorch 2.3.1, Transformers 4.41.2, Datasets 2.17.1, Tokenizers 0.19.1, Weights & Biases logging.
• Examples and evaluation in this card: Transformers 5.15.0, PyTorch 2.13.0.

Citation

Related paper from the same project:

@article{schneider2026ideology,
  title     = {Ideology Prediction of German Political Texts},
  author    = {Schneider, Sinclair and Steuber, Florian and Schneider, Jo{\~a}o A. G. and Dreo Rodosek, Gabi},
  journal   = {Proceedings of the International AAAI Conference on Web and Social Media},
  volume    = {20},
  number    = {1},
  pages     = {2006--2021},
  year      = {2026},
  doi       = {10.1609/icwsm.v20i1.42735},
  url       = {https://doi.org/10.1609/icwsm.v20i1.42735}
}

This model:

@misc{schneider_english_politic_roberta_large,
  author       = {Schneider, Sinclair},
  title        = {english\_politic\_RoBERTa-large: a RoBERTa-large classifier for detecting political text in English},
  year         = {2024},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/SinclairSchneider/english_politic_RoBERTa-large}}
}

Base model:

@article{liu2019roberta,
  title   = {RoBERTa: A Robustly Optimized BERT Pretraining Approach},
  author  = {Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal = {arXiv preprint arXiv:1907.11692},
  year    = {2019}
}

Out-of-domain evaluation data:

@inproceedings{greene2006bbc,
  title     = {Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering},
  author    = {Greene, Derek and Cunningham, P{\'a}draig},
  booktitle = {Proceedings of the 23rd International Conference on Machine Learning (ICML)},
  pages     = {377--384},
  year      = {2006}
}

Model card authors and contact

Sinclair Schneider, NLP Research Group UniBW. Questions and issues are welcome in the Community tab.