paperguard-ai-detector

PaperGuard AI Detector (v2.0)

Fine-tuned DistilBERT (distilbert-base-cased) sequence classifier that detects AI-generated text in academic papers, essays, and reports. It is the core AI-detection engine of the PaperGuard multi-agent academic-integrity system.

Labels

0 = ai, 1 = human (see config.json id2label).

Training data

v2.0 ("mega") continues from the v1.5 checkpoint and adds a larger, more diverse mix so the model sees many modern LLM writing styles:

• Claude Opus distillation — recent Anthropic-style generations
• Ateeqq/AI-and-Human-Generated-Text — academic abstracts (human + AI)
• artem9k/ai-text-detection-pile — large open human-vs-AI corpus
• (v1.5 lineage) Defactify + Ateeqq, spanning 30+ frontier LLMs (GPT-4o, LLaMA-3, Claude 3, Gemini, Mistral, Qwen, ChatGPT, …)

Important: use the logit margin, not the raw softmax

On easy/separable data the model becomes overconfident — its softmax saturates (it can report ~0% AI even on genuine AI text). The discriminative signal lives in the logit margin (human_logit − ai_logit). PaperGuard therefore scores AI-likelihood from a logistic calibration of the margin, not the raw softmax. After calibration it flags clean/academic AI at ~70–90% while keeping human text low (~10%).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tok = AutoTokenizer.from_pretrained("vediumsameer/paperguard-ai-detector")
model = AutoModelForSequenceClassification.from_pretrained("vediumsameer/paperguard-ai-detector")

text = "The rapid advancement of artificial intelligence has transformed modern education..."
inputs = tok(text, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits[0]

# Recommended: score off the margin (softmax is saturated)
margin = float(logits[model.config.label2id["human"]] - logits[model.config.label2id["ai"]])
# lower margin  -> more AI-like ; higher margin -> more human-like
print("logit margin (human - ai):", margin)

Limitations

• Blind spot: slang / style-masked AI (an LLM told to write casually) can still read as human. PaperGuard mitigates this with embedding-based stylometric "patchwork" detection, and a v2.1 trained on adversarial / multi-model / reasoning data is planned to close the gap.
• AI detection is a probabilistic indicator, not proof of authorship.

Part of PaperGuard

This model powers the AI-detection layer of PaperGuard, which also does citation claim verification, plagiarism, and writing-quality analysis. See the project repo.