hw1-hc3-detector-v1

HC3 Human vs. ChatGPT Text Detector

This model is a binary text classifier fine-tuned to distinguish between human-written text and ChatGPT-generated text.

The model was trained and evaluated on data derived from the HC3 dataset as part of an NLP classification assignment.

Model Details

Model Description

• Developed by: kunkun2
• Model type: Transformer-based binary text classifier
• Framework: PyTorch and Hugging Face Transformers
• Language: English
• Task: Human-written vs. ChatGPT-generated text classification
• Repository: kunkun2/hw1-hc3-detector-v1

The model takes a piece of English text as input and predicts whether the text is more likely to have been written by a human or generated by ChatGPT.

Uses

Direct Use

The model can be used with the Hugging Face text-classification pipeline:

from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="kunkun2/hw1-hc3-detector-v1"
)

text = "Enter the text that you want to classify here."
result = classifier(text)

print(result)

Alternatively, it can be loaded with AutoTokenizer and AutoModelForSequenceClassification:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "kunkun2/hw1-hc3-detector-v1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

text = "Enter the text that you want to classify here."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    padding=True
)

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class_id = logits.argmax(dim=-1).item()
predicted_label = model.config.id2label[predicted_class_id]

print(predicted_label)

The exact label-to-class mapping can be inspected with:

print(model.config.id2label)
print(model.config.label2id)

Intended Use

This model is intended for:

• Educational demonstrations of transformer fine-tuning
• Experiments involving human and AI-generated text classification
• Reproducing the results of the HC3 homework assignment
• Research prototypes that study AI-generated text detection

Out-of-Scope Use

This model should not be used as the sole basis for:

• Academic misconduct accusations
• Disciplinary or employment decisions
• Legal or forensic conclusions
• Moderation decisions with serious consequences
• Determining authorship with certainty

The prediction represents a statistical estimate, not proof of authorship.

Training Details

Training Data

The model was trained using data derived from the HC3 dataset, which contains human-written and ChatGPT-generated answers.

The dataset was split into:

Training Procedure

The text was tokenized with the model tokenizer and processed using truncation and padding. A pretrained transformer sequence-classification model was then fine-tuned for binary classification.

The training procedure used the training split for parameter updates and the validation split for evaluation during model development. Final performance was measured on the held-out test split.

Evaluation

Metric

Accuracy was used as the primary evaluation metric:

[ \text{Accuracy}

\frac{\text{Number of correct predictions}} {\text{Total number of predictions}} ]

Results

The fine-tuned model achieved a test accuracy of 98.91%, compared with 84.49% for the baseline.

The fine-tuned result also exceeds the assignment target accuracy of 0.90.

Bias, Risks, and Limitations

AI-generated text detection is inherently uncertain. Important limitations include:

• The model was evaluated on a particular dataset and data split.
• Performance may be lower on topics, writing styles, languages, and model families that differ from the training data.
• Newer language models may produce text that is harder to detect.
• Short, edited, translated, or paraphrased text may be misclassified.
• Human writing may be incorrectly classified as AI-generated.
• AI-generated writing may be incorrectly classified as human-written.
• Dataset artifacts may allow the model to learn patterns that do not generalize to real-world text.
• High test accuracy does not guarantee equivalent real-world performance.

Recommendations

Predictions should be treated as supporting evidence only. For consequential decisions, users should combine the model output with human review, contextual information, and additional verification methods.

The model should be evaluated on data from the intended deployment domain before practical use.

Reproducibility

The reported results were produced using the training and evaluation code submitted separately through Canvas.

• Baseline test accuracy: 0.8449
• Fine-tuned test accuracy: 0.9891
• Training examples: 37,334
• Validation examples: 4,666
• Test examples: 4,668

Citation

If you use the HC3 dataset, please cite its original paper:

@article{guo2023hc3,
  title={How Close is ChatGPT to Human Experts? Comparison Corpus,
         Evaluation, and Detection},
  author={Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi
          and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng},
  journal={arXiv preprint arXiv:2301.07597},
  year={2023}
}

Disclaimer

This model was created for educational purposes. Its predictions should not be interpreted as definitive evidence that a text was written by a human or generated by an AI system.