hw1-hc3-detector

HC3 human / ChatGPT answer classifier

Educational CS546 Homework 1 experiment. Labels: 0 = human, 1 = ChatGPT.

Training and evaluation

HC3 English revision 4d0ff18143b5a7e1b1e79beb540c04549d1e59d3. Use the first nonempty answer per class per eligible question, exclude duplicate questions and identical pairs, and split by question 80/10/10 with seed 42 before flattening the answers. Answer-only inputs; maximum length 256 wordpieces; dynamic padding; batch size 32; AdamW at 2e-5 for five epochs; weight decay 0.01. All transformer weights are fine-tuned with a sequence-classification head. Validation is monitored after each epoch. The final five-epoch model is evaluated.

Baseline test accuracy: 0.844901

Fine-tuned test accuracy: 0.989503

Fine-tuned macro F1: 0.989502

Limitations

This historical, balanced benchmark does not establish reliability on current models, new domains, multilingual text, edited text, or student submissions. Style and collection artifacts may drive predictions. Long answers are truncated. Do not use this model to make high-stakes authorship or misconduct decisions.

Source

Guo et al. (2023), How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. https://arxiv.org/abs/2301.07597