climate-fallacy-roberta

Climate Logical Fallacy Classifier (DistilRoBERTa)

This model is a DistilRoBERTa–based text classification model fine-tuned to detect logical fallacies in climate-related text.
It predicts one of 11 logical fallacy labels (including “NO_FALLACY”) for a given sentence or short paragraph.

The model was trained as part of an academic NLP project on “Automated Detection of Logical Fallacies in Climate Change Social Media Posts using Small Language Models (SLMs)”.

Model Details

• Base model: distilroberta-base
• Architecture: DistilRoBERTa (Transformer encoder, 6 layers)
• Task: Multi-class text classification
• Number of classes: 11
• Language: English
• Framework: Transformers

Label Set

The model is trained to predict the following labels:

1. CHERRY_PICKING
2. EVADING_THE_BURDEN_OF_PROOF
3. FALSE_ANALOGY
4. FALSE_AUTHORITY
5. FALSE_CAUSE
6. HASTY_GENERALISATION
7. NO_FALLACY
8. POST_HOC
9. RED_HERRINGS
10. STRAWMAN
11. VAGUENESS

id2label / label2id mappings are stored in the model config and are consistent with the training code.

📚 Training Data

The model was fine-tuned on the climate subset of the open-source dataset from:

Tariq60 – fallacy-detection repository
https://github.com/Tariq60/fallacy-detection

Only the climate portion of the dataset was used, with the standard split:

• train/ – training examples
• dev/ – validation examples
• test/ – held-out evaluation set

Each example includes:

• The climate-related text segment
• A manually assigned fallacy label (or No fallacy)

Preprocessing

• Texts were lower-cased and cleaned using a light basic_clean function:
  • Stripping extra whitespace
  • Normalising some punctuation
• Some classes were minority labels (few examples), so basic class balancing was applied via up-sampling in the training set.
• NaN or empty texts were dropped before training.

Training Procedure

• Base model: distilroberta-base
• Optimizer: AdamW (via Trainer)
• Learning rate: 2e-5
• Batch size: 16
• Max sequence length: 128–256 tokens
• Epochs: 10
• Weight decay: 0.01
• Loss function: Cross-entropy, optionally with class weights to mitigate class imbalance
• Validation split: 80/20 stratified split of the training data

Implementation used:

• AutoTokenizer
• AutoModelForSequenceClassification
• TrainingArguments
• Trainer

from the Transformers library.

Evaluation

Evaluation was done on the held-out climate test set from the dataset.

Metrics (multi-class):

• Accuracy ≈ 0.24
• Macro F1 ≈ 0.20
• Weighted F1 ≈ 0.24

These values are baseline experimental results on a relatively small and imbalanced dataset. They should be interpreted as preliminary research numbers, not as production-ready performance.

Different random seeds, data balancing strategies, or more aggressive hyperparameter tuning can change these numbers.

Intended Use

Primary Use

• Research and experimentation on:
  • Automated detection of logical fallacies in climate discourse
  • Comparing traditional baselines (TF-IDF + SVM) vs. Transformer-based models
  • Building educational tools that flag potential fallacies in climate arguments

Suitable Scenarios

• Analyzing short climate-related social media posts
• Demonstration / teaching examples on:
  • Argumentation quality
  • Climate misinformation
  • Explainable NLP (combined with a small language model explainer, e.g. FLAN-T5)

Limitations & Ethical Considerations

Limitations

• Small dataset: Training data is limited in size, especially for rarer fallacy types.
• Class imbalance: Some fallacies occur far less frequently, which affects per-class F1 scores.
• Modest performance: Overall accuracy and macro F1 are relatively low. The model should be treated as an exploratory research artifact, not a production system.
• Domain specificity: The model is trained only on climate discourse; performance on other topics (e.g. politics, health) is unknown and likely poor.

Ethical Considerations

• Predictions are probabilistic, not definitive judgments of truth or deception.
• The model can be wrong or over-confident, especially on borderline or nuanced arguments.
• It should not be used for automated moderation, censorship, or any high-stakes decision-making without strong human oversight.

How to Integration with Explanatory SLM

In the associated project, this classifier is combined with a small language model (e.g., google/flan-t5-small) to generate natural-language explanations of the predicted fallacy label:

What the fallacy means in simple terms

Why the input text might be an example

This setup is used in a Streamlit app:

Users enter a climate-related argument

The model predicts a fallacy label

FLAN-T5 generates a short explanation

Citation

If you use this model in academic work, you can cite it as:

Kyeremeh, F. (2025). Climate Logical Fallacy Classifier (DistilRoBERTa). Hugging Face. Model: SteadyHands/climate-fallacy-roberta.

And also consider citing the original dataset author(s):

Tariq60. fallacy-detection GitHub repository. https://github.com/Tariq60/fallacy-detection

Acknowledgements

Base model: distilroberta-base by Hugging Face

Dataset: Climate subset from Tariq60’s fallacy-detection repository

Libraries:

Transformers

Datasets

scikit-learn

Project context:

Master ’s-level NLP / Data Science coursework on Small Language Models and explainable NLP.

How to Use

Python Example (Logits → Label)

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "SteadyHands/climate-fallacy-roberta"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Climate has always changed in the past, so current warming can't be caused by humans."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding="max_length",
    max_length=256,
)

with torch.no_grad():
    outputs = model(**inputs)

logits = outputs.logits
probs = torch.softmax(logits, dim=-1)[0].tolist()
pred_id = int(torch.argmax(logits, dim=-1).item())

id2label = model.config.id2label
pred_label = id2label[str(pred_id)] if isinstance(id2label, dict) else id2label[pred_id]

print("Text:", text)
print("Predicted label:", pred_label)
print("Probabilities:", probs)


Using the Transformers Pipeline

```python
from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="SteadyHands/climate-fallacy-roberta",
    top_k=None,  # set top_k=3 to see top-3 fallacies
)

text = "Temperatures dropped this winter, so global warming must be a hoax."
outputs = clf(text)

print(outputs)