claimbuster-detector

ClaimBuster Detector

A DistilBERT model fine-tuned to decide whether a sentence is a check-worthy factual claim: a statement the public would care about and that can be verified, such as a statistic, a historical fact or a claim about an organization.

It is the claim detection stage of DebateIQ Pakistan, an AI platform that analyzes speeches and debates for credibility. In that pipeline, this model picks out the sentences worth fact-checking, and later stages retrieve evidence and verify them.

Model details

Labels

How to use

from transformers import pipeline

clf = pipeline("text-classification", model="sachaljawad/claimbuster-detector")

clf("The World Bank reported 70% youth unemployment.")
# [{'label': 'checkworthy', 'score': ...}]

clf("Thank you all for coming today.")
# [{'label': 'not_checkworthy', 'score': ...}]

To get the probability that each sentence is check-worthy:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo = "sachaljawad/claimbuster-detector"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

sentences = ["Pakistan won the cricket world cup in 1992.", "I had coffee this morning."]
enc = tok(sentences, return_tensors="pt", truncation=True, padding=True, max_length=128)
with torch.no_grad():
    probs = torch.softmax(model(**enc).logits, dim=-1)[:, 1]
print(probs.tolist())

Intended uses

Good fits

• The first stage of a fact-checking pipeline: filtering a transcript down to the sentences worth verifying
• Highlighting factual claims in speeches, debates, interviews and articles
• Research and education on claim detection

Not intended for

• Deciding whether a claim is true or false. The model only judges whether a sentence is worth checking.
• Non-English text. Translate first (DebateIQ translates Urdu to English before this step).
• Fully automated moderation or decisions about people without human review

Training data

Fine-tuned on the ClaimBuster dataset (Arslan et al., 2020), which contains sentences from U.S. presidential debates labelled by human annotators into three classes:

• CFS (Check-worthy Factual Sentence)
• UFS (Unimportant Factual Sentence)
• NFS (Non-Factual Sentence)

These were mapped to a binary task: CFS → checkworthy (1), and UFS and NFS → not_checkworthy (0).

Preprocessing:

• Removed empty rows and sentences of 5 characters or fewer
• Removed duplicate sentences
• Stratified 90/10 train/validation split (random_state=42)

Training procedure

Evaluation

Evaluated on the held-out 10% stratified validation split (~2,240 sentences).

F1 is reported for the positive (checkworthy) class, which is the minority class, so it is a stricter measure than accuracy.

Limitations and bias

• Domain shift. The training data comes from U.S. presidential debates. Performance may drop on other settings, such as Pakistani political speech, student debates or casual conversation, and on region-specific names and topics.
• Check-worthiness is subjective. Annotators can disagree about what deserves checking, and the model inherits those judgements.
• Sentence-level only. Claims that depend on context from earlier sentences may be missed.
• Uncased and English only. Translated text may lose nuance before it reaches the model.

Treat the outputs as suggestions for human fact-checkers, not final judgements.

Citation

If you use this model, please also cite the ClaimBuster dataset:

@inproceedings{arslan2020benchmark,
  title     = {A Benchmark Dataset of Check-worthy Factual Claims},
  author    = {Arslan, Fatma and Hassan, Naeemul and Li, Chengkai and Tremayne, Mark},
  booktitle = {Proceedings of the International AAAI Conference on Web and Social Media},
  volume    = {14},
  pages     = {821--829},
  year      = {2020}
}

Contact

Mohammad Sachal Jawad · GitHub · LinkedIn