Political_DEBATE_large_v1.0

Model Details

Political DEBATE (DeBERTa Algorithm for Textual Entailment) is an NLI classifier trained for zero-shot and few-shot classification of political documents.

Zero-shot tutorial
Few-shot tutorial

The model was trained using Moritz Laurer's zero shot model as the foundation, and then trained further using the PolNLI dataset. The model was trained for four classification categories:

1. Stance detection (i.e. opinion mining).
2. Hatespeech detection.
3. Event extraction.
4. Topic classification.

The PolNLI dataset contains documents from social media, news outlets, congressional bills, court case summaries, and more. Classification should work well across a broad set of document sources and subjects, but for best performance refer to the recommendations section.

If you use this model or the PolNLI data set please cite:

@article{burnham2024political,
  title={Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text},
  author={Burnham, Michael and Kahn, Kayla and Wang, Ryan Yank and Peng, Rachel X},
  journal={arXiv preprint arXiv:2409.02078},
  year={2024}
}

Using NLI Classifiers

NLI classifiers work by pairing a document (AKA the 'premise') with a 'hypothesis', and determining if the hypothesis is true given the content of the document. The hypothesis is supplied by the user and can be thought of as a simplified version of a prompt for an LLM. It's best to keep hypotheses short and simple with a single classification criteria. If a more complicated hypothesis is necessary, consider few-shot training. For more detailed reading on using NLI classifiers see:

@article{burnham2024stance,
  title={Stance detection: a practical guide to classifying political beliefs in text},
  author={Burnham, Michael},
  journal={Political Science Research and Methods},
  year={2024},
  publisher={Cambridge University Press}
}

Recommendations

1. Use the Political DEBATE large model for zero-shot and few-shot applications unless your use case is explicitly in the training data, in which case the base model should perform well. The base model is more appropriate for using the model as a supervised classifier.
2. For best results, format hypotheses similar to the PolNLI dataset. Ex:

• "This text is about {}"
• "The author of this text believes {}"

1. For few-shot training, use a small batch size of 1-2.
2. Use a minimum of 10 samples for few-shot training, but we generally recommend 25 because the variance of outcomes at 10 can be quite large. You will see more benefit from more samples the more difficult your classification task and varied your data.
3. Train for 3-5 epochs for few-shot learning.

Evaluation

Evaluation is primarily conducted on the PolNLI test set. No hypotheses in the PolNLI test set are present in the training data in order to estimate zero-shot performance.

Overal performance on the PolNLI test set:

<img src="https://cdn-uploads.huggingface.co/production/uploads/64d0341901931c60161f2a06/-sQhURg-zeacZAUNEVUpc.png" width="750" height="500" />

Performance on the PolNLI test set by task:

<img src="https://cdn-uploads.huggingface.co/production/uploads/64d0341901931c60161f2a06/ss6cN8ujGl40SZ5WxjiDH.png" width="750" height="500" />

Few-shot performance on open-text survey answers vs. Llama 3.1 8B (unquantized):

<img src="https://cdn-uploads.huggingface.co/production/uploads/64d0341901931c60161f2a06/kDZXC-OsZtmTCxq0HsJMg.png" width="750" height="500" />

Few-shot performance vs. and Electra classifier trained on 2,000 documents:

<img src="https://cdn-uploads.huggingface.co/production/uploads/64d0341901931c60161f2a06/WpQ_ZofMJMFPraCK_a3Fb.png" width="750" height="500" />