fake-news-bert-debiased

Fake News Detector — BERT (Debiased V2)

Model Description

This model is a fine-tuned version of bert-base-uncased for binary fake news classification (real vs. fake), trained on a cleaned version of the WELFake dataset with a specific shortcut-learning bug fixed. It supersedes fake-news-bert, which achieved similar benchmark accuracy but for the wrong reasons — see below.

• Developed by: Lax (LakshmiNarayanan Sugumar)
• Model type: BERT-base, fine-tuned for sequence classification
• Language: English
• License: Apache 2.0 (inherited from base model)
• Finetuned from: bert-base-uncased
• Repository: github.com/LakshmiNarayanan-Sugumar/fake_news_detector_v2
• Demo: HuggingFace Space — fake-news-detector

Why This Model Exists: A Shortcut-Learning Bug

The original model (fake-news-bert) hit 99.53% test accuracy but performed poorly on real-world text outside the WELFake distribution — especially satire and hoaxes. A hand-built 26-example out-of-distribution (OOD) test set (satire, hoaxes, and real news from diverse sources) exposed this: overall OOD accuracy was only 53.8%.

Investigating further, a quantitative diagnostic on the training data revealed the cause: 61.8% of "real" articles in WELFake mentioned "Reuters," versus only 1.8% of "fake" articles. The model had learned to key off this single word rather than genuine signals of factuality — an easy, cheatable shortcut that inflated benchmark accuracy while masking real generalization failure. Secondary shortcut signals (ALL-CAPS phrases, excessive exclamation marks) were also present.

What Changed

Training text was cleaned to remove these shortcuts: "Reuters" mentions stripped, ALL-CAPS phrases collapsed to normal case, and exclamation marks capped. The model was retrained for 2 epochs on this cleaned data.

Before / After OOD Comparison

The fix worked as intended: real-news accuracy on non-Reuters-sourced articles went from 87.5% to 100%, confirming the shortcut is gone and the model now identifies real news on genuine content signals rather than source-name pattern-matching.

A known limitation remains: satire and hoax detection did not improve, and hoax accuracy on the small sample slightly declined. This is not a regression from the fix — it reflects a genuine data diversity gap. WELFake's "fake" class is dominated by aggressively-toned misinformation and contains little dry-toned satire or calmly-written hoaxes, so the model never learned to recognize that register as suspicious. Every misclassified OOD example was predicted "real" with high confidence (92–99%), indicating the model isn't confused — it's simply never seen this writing style labeled as fake before. Fixing this requires more diverse training data (e.g. incorporating ISOT or other satire-inclusive sources), not further text cleaning.

Uses

Direct Use

Given a news article's title and body text, the model predicts whether the article is real or fake. Intended for educational and portfolio demonstration purposes.

Out-of-Scope Use

Not intended for production moderation or fact-checking decisions without human review. As documented above, the model has a known weakness on satire and dry-toned hoaxes.

How to Get Started

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("LakshmiNarayanan-sugumar/fake-news-bert-debiased")
model = AutoModelForSequenceClassification.from_pretrained("LakshmiNarayanan-sugumar/fake-news-bert-debiased")

text = "Your article title here. Your article body text here."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
prediction = torch.argmax(logits, dim=-1).item()
print("Fake" if prediction == 1 else "Real")

Training Details

Training Data

WELFake dataset, cleaned to 62,200 deduplicated articles, split 80/10/10 (train/val/test, stratified). Training text additionally cleaned of shortcut signals (Reuters mentions, ALL-CAPS, excessive exclamation marks) as described above.

Training Procedure

Title and body concatenated into a single input, tokenized with the BERT tokenizer (max length 512, truncation enabled). Fine-tuned for 2 epochs, batch size 16.

Training Hyperparameters

• Epochs: 2
• Batch size: 16 (train and eval)
• Base model: bert-base-uncased
• Hardware: Kaggle T4 GPU

Evaluation

Test Set Results (held-out WELFake split, 6,220 articles)

Near-identical to the original model's in-distribution metrics (99.53% / 99.47% / 99.64% / 99.31%), confirming that removing the shortcut cost negligible in-distribution performance — the model was already learning real signal alongside the shortcut.

Out-of-Distribution Test Set

See the before/after table above. Full 26-example OOD test set (ood_test.csv) available in the GitHub repository.

Technical Specifications

• Architecture: BertForSequenceClassification (2 labels), built on bert-base-uncased
• Compute: Kaggle Notebooks, T4 GPU

Model Card Contact

LakshmiNarayanan Sugumar — GitHub