distilbert-hc3-human-vs-ai

DistilBERT HC3 Human vs AI Detector

This is a fully fine-tuned DistilBERT binary sequence classifier for distinguishing human-written answers from AI-generated answers. It was trained on the English all configuration of Hello-SimpleAI/HC3, using answer text only.

The data pipeline isolates the original HC3 questions across train, validation, and test splits before balancing. Consequently, answers derived from the same question cannot appear in more than one split.

Important: this model measures similarity to the human and early-ChatGPT writing patterns represented in HC3. Its output is a model score—not proof of authorship. Do not use it alone for grading, discipline, hiring, moderation, or accusations of AI use.

Model details

Label mapping

Held-out test results

The final model was evaluated once on a balanced, question-group-isolated test set containing 5,306 examples: 2,653 human answers and 2,653 AI-generated answers.

Per-class results

Confusion matrix

Rows are true labels; columns are predicted labels.

These results describe the held-out HC3 split only. They must not be interpreted as expected performance on newer language models, edited text, other languages, or unrelated domains.

Run test cells

The following cells load the published checkpoint, run single/batch inference, return both class probabilities, and perform a small smoke test.

Cell 1 — Install dependencies

%pip install -q "transformers==4.57.1" "accelerate>=1.2,<2"

If the notebook runtime asks for a restart after installation, restart it once before continuing.

Cell 2 — Load the model

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "AyoubChLin/distilbert-hc3-human-vs-ai"
MAX_LENGTH = 512

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.to(device)
model.eval()

print(f"Loaded {MODEL_ID} on {device}")
print("Label mapping:", model.config.id2label)

Cell 3 — Reliable inference function

def predict_texts(texts, batch_size=16, ai_threshold=0.50):
    """Classify one string or a list of strings.

    `ai_threshold` applies to the AI_GENERATED score. The default 0.50
    threshold is suitable for a basic demo; recalibrate it on representative
    in-domain validation data before deploying the model.
    """
    if isinstance(texts, str):
        texts = [texts]

    if not isinstance(texts, list) or not texts:
        raise ValueError("texts must be a non-empty string or list of strings")
    if not all(isinstance(text, str) and text.strip() for text in texts):
        raise ValueError("every input must be a non-empty string")
    if not 0.0 <= ai_threshold <= 1.0:
        raise ValueError("ai_threshold must be between 0 and 1")

    ai_id = int(model.config.label2id.get("AI_GENERATED", 1))
    human_id = int(model.config.label2id.get("HUMAN", 0))
    results = []

    for start in range(0, len(texts), batch_size):
        batch = texts[start : start + batch_size]
        inputs = tokenizer(
            batch,
            padding=True,
            truncation=True,
            max_length=MAX_LENGTH,
            return_tensors="pt",
        ).to(device)

        with torch.inference_mode():
            probabilities = model(**inputs).logits.softmax(dim=-1).cpu()

        for text, probs in zip(batch, probabilities):
            human_score = float(probs[human_id])
            ai_score = float(probs[ai_id])
            prediction = "AI_GENERATED" if ai_score >= ai_threshold else "HUMAN"

            results.append({
                "text": text,
                "prediction": prediction,
                "confidence": max(human_score, ai_score),
                "human_probability": human_score,
                "ai_probability": ai_score,
                "truncated_to_max_length": len(
                    tokenizer(text, add_special_tokens=True)["input_ids"]
                ) > MAX_LENGTH,
            })

    return results

Cell 4 — Single-text test

result = predict_texts(
    "I tried the recipe yesterday. It was a little too salty, "
    "but my family still finished everything."
)[0]

print(f"Prediction:       {result['prediction']}")
print(f"Confidence:       {result['confidence']:.4f}")
print(f"Human score:      {result['human_probability']:.4f}")
print(f"AI score:         {result['ai_probability']:.4f}")
print(f"Input truncated:  {result['truncated_to_max_length']}")

Cell 5 — Batch test with validation checks

samples = [
    "I missed the bus this morning, so I walked to work in the rain.",
    "Machine learning is a field of study that enables systems to learn patterns from data and make predictions.",
    "The first version failed twice. I changed the parser, reran it, and finally got the output I expected.",
]

results = predict_texts(samples, batch_size=8)

assert len(results) == len(samples)
for item in results:
    assert item["prediction"] in {"HUMAN", "AI_GENERATED"}
    assert 0.0 <= item["confidence"] <= 1.0
    assert abs(
        item["human_probability"] + item["ai_probability"] - 1.0
    ) < 1e-5

    print("-" * 80)
    print(f"Prediction: {item['prediction']} | confidence={item['confidence']:.4f}")
    print(f"HUMAN={item['human_probability']:.4f} | AI_GENERATED={item['ai_probability']:.4f}")
    print(item["text"])

print("\nSmoke test passed.")

The sample labels are intentionally not asserted: an inference smoke test should verify that the checkpoint loads and returns valid probabilities, not treat a few hand-written sentences as ground truth.

Cell 6 — Optional Transformers pipeline

from transformers import pipeline

classifier = pipeline(
    task="text-classification",
    model=MODEL_ID,
    tokenizer=MODEL_ID,
    device=0 if torch.cuda.is_available() else -1,
)

classifier(
    "Paste the passage you want to inspect here.",
    truncation=True,
    max_length=512,
)

Training data

HC3 contains questions paired with lists of human answers and answers generated by an early ChatGPT system. The all configuration combines five English sources:

• finance
• medicine
• open_qa
• reddit_eli5
• wiki_csai

Cleaning and leakage prevention

The preparation pipeline used the following procedure:

1. Expanded every question row into standalone human and AI answer records.
2. Normalized repeated whitespace and removed answers shorter than 20 characters.
3. Compared normalized, lower-cased text to remove exact duplicates globally.
4. Removed ambiguous text that appeared under both labels.
5. Assigned every answer from the same original question a shared group_id.
6. Split question groups 80/10/10, stratified by source, before balancing.
7. Balanced HUMAN and AI_GENERATED independently inside each split.

There were 79,325 usable unique answers before per-split balancing.

Training procedure

The classification head was initialized for the downstream binary task, and all model parameters were updated.

The configuration selected batch sizes from available GPU memory:

Recorded training statistics

The notebook requested up to five epochs and reached epoch 3.0 with early stopping enabled. The final exported checkpoint was loaded from the best validation-F1 checkpoint.

Intended use

This model is suitable for:

• research and educational experiments on HC3-style AI-text detection;
• benchmarking binary text-classification pipelines;
• exploratory screening where every result is reviewed by a human;
• serving as a baseline before training or calibrating on newer, in-domain data.

It is not suitable as a standalone authorship verifier or as the sole basis for consequential decisions.

Limitations and failure modes

• Dataset age: HC3's synthetic class represents an early ChatGPT system, not the full range of current generators.
• Domain shift: performance can fall sharply on domains and writing styles that are not represented in HC3.
• English only: the model and training corpus are English-focused. Results for other languages are unsupported.
• Editing and paraphrasing: human editing, paraphrasing, translation, deliberate evasion, or mixed human/AI authorship can change the prediction substantially.
• False positives and false negatives: polished human prose may resemble the AI class, while generated text may resemble the human class.
• Short text: short passages provide little stylistic evidence and are intrinsically harder to classify.
• Long text: inputs beyond 512 tokens are truncated; the default score therefore does not represent the entire document.
• Scores are not calibrated proof: softmax values are confidence scores under this model and dataset. They are not the real-world probability that an author used AI.
• Balanced evaluation: the test set is class-balanced. Precision and predictive value will change when the real deployment prevalence differs.
• No question context: the model receives only answer text, even though the original HC3 records also contain questions.

For deployment, collect recent human and AI examples from the target domain, preserve author/source groups across splits, evaluate subgroup error rates, calibrate the threshold, and monitor drift and false positives continuously.

Ethical considerations

AI-text detectors can incorrectly accuse people, and stylistic differences may produce uneven error rates across writers, language backgrounds, accessibility needs, and levels of writing experience. Predictions should be treated as weak supporting signals and reviewed alongside transparent, independent evidence. Users should have a meaningful way to contest any consequential decision.

Reproducibility notes

• Random seed and data seed: 42
• Full balanced training corpus: MAX_SAMPLES_PER_CLASS = None
• Tokenization: fast DistilBERT tokenizer, truncation at 512 tokens
• Evaluation: binary precision/recall/F1 for AI_GENERATED, plus accuracy and ROC AUC
• Held-out evaluation split: 5,306 examples, isolated at the original-question level

Citation

If you use this checkpoint, cite the HC3 dataset and the DistilBERT work in addition to referencing this model repository.

@inproceedings{guo2023hc3,
  title     = {How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author    = {Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year      = {2023}
}

@inproceedings{sanh2019distilbert,
  title     = {DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter},
  author    = {Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas},
  booktitle = {NeurIPS EMC2 Workshop},
  year      = {2019}
}

License

This model repository is released under CC BY-SA 4.0. Users are responsible for complying with the licenses and terms of the base model and HC3 dataset.