hw1-hc3-detector

HW1 HC3 Detector (MiniLM-L6 fine-tuned)

A binary classifier that predicts whether an answer from the HC3 dataset was written by a human (label 0) or by ChatGPT (label 1). It was built for a course homework (CS546 HW1) on adapting pretrained language models.

Results (HC3 test split, 4,668 answers)

Confusion matrix of the fine-tuned model on the test split (rows are true labels, columns are predictions):

Training

• Base model: sentence-transformers/all-MiniLM-L6-v2, loaded with AutoModelForSequenceClassification (2 labels, cross-entropy)
• Data: Hello-SimpleAI/HC3 (English, all.jsonl, revision 4d0ff18). The first non-empty human answer and the first non-empty ChatGPT answer were kept for each question. The questions were split 80/10/10 with seed 42, so both answers to a question stay in the same split. Train/val/test sizes are 37,334 / 4,666 / 4,668 answers, balanced between the two classes.
• Input: answer text only, truncated to 256 tokens, with dynamic padding
• Optimizer: AdamW, learning rate 2e-5, batch size 32, 5 epochs (5,835 steps), no scheduler, seed 42

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "JacobWu123/hw1-hc3-detector"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

inputs = tok("Your answer text here", truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]
print({"human": probs[0].item(), "ChatGPT": probs[1].item()})

Limitations

• The 99% accuracy is in-distribution: the training and test splits share the same sources (about 70% Reddit ELI5), the same ChatGPT version (December 2022) and the same preprocessing. The model may rely on style cues specific to this dataset, such as Reddit register, answer length or 2022-era ChatGPT phrasing.
• Nearly all remaining errors are human answers flagged as ChatGPT.
• Do not use this model to judge whether real people, such as students, used AI. It is not a reliable detector for text from other domains or from newer language models.