bert-hc3-human-vs-ai

BERT HC3 Human vs AI Text Detector

This model is a full fine-tune of google-bert/bert-base-uncased for binary classification of standalone English passages as human-written or AI-generated.

It was trained on answer text from the Hello-SimpleAI/HC3 corpus. The data pipeline performs global exact-text deduplication, removes texts found under both labels, splits by the original HC3 question before balancing, and keeps all answers associated with a question in one split. This avoids direct question-level leakage between training, validation, and test data.

Important: this is a similarity detector, not an authorship verifier. A high AI_GENERATED score means that the input resembles HC3's early-ChatGPT examples under this model. It is not proof that a text was produced by AI.

Model details

Labels

id2label = {0: "HUMAN", 1: "AI_GENERATED"}
label2id = {"HUMAN": 0, "AI_GENERATED": 1}

Intended use

This checkpoint is suitable for:

• research and educational experiments on HC3-style AI-text detection;
• benchmarking a binary text-classification pipeline;
• testing domain shift, calibration, robustness, and false-positive behavior;
• use as a baseline before fine-tuning on recent, in-domain human and AI text.

It should not be used by itself for grading, punishment, hiring, moderation, academic-integrity accusations, legal decisions, or any other consequential authorship judgment.

Training data

HC3 contains questions and lists of human and ChatGPT answers from five English sources: finance, medicine, open_qa, reddit_eli5, and wiki_csai.

The run loaded all.jsonl directly because current datasets releases do not execute the repository's legacy dataset script.

Data preparation

1. Loaded 24,322 HC3 question rows.
2. Expanded every human and ChatGPT answer into one classification example.
3. Collapsed whitespace and removed answers shorter than 20 characters.
4. Compared normalized lowercase text globally.
5. Removed any text occurring under both labels, then removed remaining exact duplicates.
6. Retained 79,325 unique usable answers: 53,082 human and 26,243 AI-generated.
7. Split unique question groups 80/10/10 with source stratification and seed 42.
8. Balanced the labels independently within every split by downsampling the larger class.

Final balanced splits

The held-out test set was not used for model selection. Validation F1 was used to select the best checkpoint.

Training procedure

The model was fully fine-tuned; this is not a LoRA, adapter, or quantized checkpoint.

The attached run reported an NVIDIA A100 80 GB PCIe with BF16 enabled. The notebook's adaptive single-GPU configuration selects train/evaluation batch sizes of 128/256 for GPUs with at least 70 GiB, 32/64 for GPUs with at least 40 GiB, and 8/16 otherwise. It was designed for A100, H100, H200, or comparable CUDA hardware.

Recorded evaluation status

The supplied notebook defines accuracy, binary precision, binary recall, binary F1, ROC AUC, a classification report, and a confusion matrix. However, its saved test cell contains no output, and no public test_metrics.json was available when this card was prepared. Exact scores are therefore intentionally not claimed.

Run the reproducible held-out evaluation cell below and publish the resulting values before using this checkpoint as a reported benchmark. Do not tune the model or its threshold after inspecting the held-out test results.

Quick start

Cell 1 — install

%pip install -q -U "transformers>=4.46,<6" "torch>=2.2" "scikit-learn>=1.4" "pandas>=2.0"

Cell 2 — load and validate the checkpoint

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "AyoubChLin/bert-hc3-human-vs-ai"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
model.to(DEVICE).eval()

assert model.config.num_labels == 2
assert model.config.id2label == {0: "HUMAN", 1: "AI_GENERATED"}
assert model.config.label2id == {"HUMAN": 0, "AI_GENERATED": 1}

print(f"Loaded {MODEL_ID} on {DEVICE}")
print(model.config.id2label)

Cell 3 — single and batched inference with probabilities

import pandas as pd

@torch.inference_mode()
def predict_texts(texts, batch_size=32):
    if isinstance(texts, str):
        texts = [texts]
    if not texts:
        return pd.DataFrame(columns=[
            "text", "prediction", "human_probability", "ai_probability"
        ])

    rows = []
    for start in range(0, len(texts), batch_size):
        batch = texts[start : start + batch_size]
        encoded = tokenizer(
            batch,
            padding=True,
            truncation=True,
            max_length=512,
            return_tensors="pt",
        ).to(DEVICE)

        probabilities = model(**encoded).logits.softmax(dim=-1).cpu()

        for text, probs in zip(batch, probabilities):
            predicted_id = int(probs.argmax())
            rows.append({
                "text": text,
                "prediction": model.config.id2label[predicted_id],
                "human_probability": float(probs[0]),
                "ai_probability": float(probs[1]),
            })

    result = pd.DataFrame(rows)

    # Contract tests validate the checkpoint without pretending that two
    # hand-written examples establish model quality.
    assert len(result) == len(texts)
    assert set(result["prediction"]).issubset({"HUMAN", "AI_GENERATED"})
    assert result[["human_probability", "ai_probability"]].ge(0).all().all()
    assert result[["human_probability", "ai_probability"]].le(1).all().all()
    assert torch.allclose(
        torch.tensor(result["human_probability"] + result["ai_probability"]),
        torch.ones(len(result)),
        atol=1e-5,
    )
    return result

examples = [
    "I tried the recipe yesterday. It was a little too salty, but my family still finished everything.",
    "Artificial intelligence is a transformative technology that can improve efficiency across a wide range of industries.",
]

display(predict_texts(examples))

Do not interpret a displayed score as a calibrated probability of real-world AI authorship. It is the checkpoint's softmax score under its learned HC3 decision boundary.

Cell 4 — Transformers pipeline

from transformers import pipeline

classifier = pipeline(
    task="text-classification",
    model=MODEL_ID,
    tokenizer=MODEL_ID,
    device=0 if torch.cuda.is_available() else -1,
)

results = classifier(
    examples,
    top_k=None,
    truncation=True,
    max_length=512,
)

for text, scores in zip(examples, results):
    print("\nTEXT:", text)
    print(sorted(scores, key=lambda row: row["score"], reverse=True))

Exact held-out test reproduction

This cell reconstructs the same deterministic held-out split from HC3 and evaluates the uploaded checkpoint. It repeats the notebook's cleaning, grouping, stratification, balancing, and seeds.

%pip install -q -U "datasets>=3.2,<5" "scikit-learn>=1.4" "pandas>=2.0" "tqdm>=4.66"

import re
import numpy as np
import pandas as pd
import torch
from datasets import load_dataset
from sklearn.metrics import (
    accuracy_score,
    classification_report,
    confusion_matrix,
    precision_recall_fscore_support,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split
from tqdm.auto import tqdm

SEED = 42
DATA_URL = "https://huggingface.co/datasets/Hello-SimpleAI/HC3/resolve/main/all.jsonl"

raw = load_dataset("json", data_files={"train": DATA_URL}, split="train")

def clean_text(text):
    if not isinstance(text, str):
        return ""
    return re.sub(r"\s+", " ", text).strip()

records = []
for group_id, row in enumerate(tqdm(raw, desc="Expanding HC3 answers")):
    for label, column in ((0, "human_answers"), (1, "chatgpt_answers")):
        for answer in row[column]:
            text = clean_text(answer)
            if len(text) >= 20:
                records.append({
                    "text": text,
                    "label": label,
                    "group_id": group_id,
                    "source": row["source"],
                })

df = pd.DataFrame(records)
df["normalized_text"] = (
    df["text"].str.lower().str.replace(r"\s+", " ", regex=True).str.strip()
)
label_counts = df.groupby("normalized_text")["label"].nunique()
ambiguous = set(label_counts[label_counts > 1].index)
df = df.loc[~df["normalized_text"].isin(ambiguous)].copy()
df = df.drop_duplicates("normalized_text", keep="first")
df = df.drop(columns="normalized_text").reset_index(drop=True)

group_table = df[["group_id", "source"]].drop_duplicates("group_id")
_, temporary_groups = train_test_split(
    group_table,
    test_size=0.20,
    random_state=SEED,
    stratify=group_table["source"],
)
_, test_groups = train_test_split(
    temporary_groups,
    test_size=0.50,
    random_state=SEED,
    stratify=temporary_groups["source"],
)

test_df = df.loc[df["group_id"].isin(set(test_groups["group_id"]))].copy()
n_per_class = int(test_df["label"].value_counts().min())
test_df = pd.concat([
    test_df.loc[test_df["label"] == label].sample(
        n=n_per_class,
        random_state=SEED + label,
    )
    for label in (0, 1)
]).sample(frac=1, random_state=SEED).reset_index(drop=True)

assert len(test_df) == 5306
assert test_df["label"].value_counts().to_dict() == {0: 2653, 1: 2653}

all_probabilities = []
batch_size = 64 if DEVICE.type == "cuda" else 8
with torch.inference_mode():
    for start in tqdm(range(0, len(test_df), batch_size), desc="Evaluating"):
        texts = test_df["text"].iloc[start : start + batch_size].tolist()
        encoded = tokenizer(
            texts,
            padding=True,
            truncation=True,
            max_length=512,
            return_tensors="pt",
        ).to(DEVICE)
        probs = model(**encoded).logits.softmax(dim=-1).cpu().numpy()
        all_probabilities.append(probs)

probabilities = np.concatenate(all_probabilities, axis=0)
labels = test_df["label"].to_numpy()
predictions = probabilities.argmax(axis=1)
precision, recall, f1, _ = precision_recall_fscore_support(
    labels, predictions, average="binary", zero_division=0
)
metrics = {
    "accuracy": accuracy_score(labels, predictions),
    "precision": precision,
    "recall": recall,
    "f1": f1,
    "roc_auc": roc_auc_score(labels, probabilities[:, 1]),
}

print(pd.Series(metrics).round(6))
print()
print(classification_report(
    labels,
    predictions,
    target_names=["HUMAN", "AI_GENERATED"],
    digits=4,
    zero_division=0,
))
print("Confusion matrix:\n", confusion_matrix(labels, predictions))

Limitations and biases

• Older generator distribution: HC3's synthetic answers come from an early ChatGPT system; newer models can differ substantially.
• English only: predictions on other languages are unsupported.
• Domain dependence: performance may drop outside the five HC3 sources.
• Length sensitivity: short or fragmentary passages contain less evidence and are generally harder to classify.
• Truncation: only the first 512 WordPiece tokens are used.
• Editable signals: paraphrasing, translation, human revision, spelling changes, prompt style, or mixed authorship can change a prediction.
• Classification errors: fluent human prose may resemble HC3's AI class, while generated text may resemble its human class.
• Uncalibrated confidence: softmax values are not real-world probabilities of authorship without representative calibration.
• Balanced evaluation: the 50/50 test distribution does not represent real-world prevalence; precision changes with class prevalence.

For deployment, build a recent in-domain evaluation set, keep author/source/prompt groups isolated, report per-domain and per-length results, calibrate the score, choose a threshold from validation data, monitor drift, and retain human review.

Ethical considerations

AI-text detectors can cause harm when uncertain predictions are presented as facts. Users should be told what the score means, what data the system was trained on, and how often it fails on representative examples. Provide an appeal or review path in consequential settings, and never use this checkpoint as the sole evidence of misconduct.

Reproducibility

The notebook used these package constraints:

transformers>=4.46,<6
datasets>=3.2,<5
accelerate>=1.2
huggingface_hub>=0.27
scikit-learn>=1.4
pandas>=2.0
matplotlib>=3.8
seaborn>=0.13

Python, NumPy, PyTorch, Transformers, split, sampling, and data seeds were fixed to 42. Exact bitwise reproducibility can still vary with package versions, CUDA/cuDNN kernels, hardware, and nondeterministic GPU operations.

Citation

If you use the dataset, cite the HC3 paper:

@article{guo2023hc3,
  title   = {How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection},
  author  = {Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng},
  journal = {arXiv preprint arXiv:2301.07597},
  year    = {2023}
}

Acknowledgements

This checkpoint builds on Google's BERT and the HC3 corpus released by Hello-SimpleAI. Review the licenses and terms of the base model, HC3, and each upstream HC3 source before redistribution or commercial use.