roberta-base-all-sources-ai-identifier

RoBERTa All-Sources AI Text Identifier

This model is a roberta-base sequence classifier fine-tuned to distinguish human-written text from AI-generated text, with emphasis on creative-writing and prose-like distributions.

Labels:

• human: label 0
• ai: label 1

Intended Use

The model is intended for research and internal evaluation of AI-generated prose detection. It should not be used as the sole basis for high-stakes decisions about authorship, academic integrity, employment, moderation, or user sanctions. Outputs are classifier probabilities, not proof of authorship.

Loading

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "frankhanhj/roberta-base-all-sources-ai-identifier"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

Training Data

The training corpus is datasets/all-sources-ai-identifier, assembled from the local sources below. Validation and test rows were not used for training. For chunked external sources, splits were isolated by group_id / original_index; the recorded split audit found 0 leaks across 10,992 checked groups.

Overall split sizes:

Techniques Tried

Several detector families were trained or evaluated before selecting this checkpoint:

The selected model is the all-sources RoBERTa checkpoint. It has slightly lower original mixed-benchmark accuracy than the earlier mixed-plus-Claude RoBERTa run, but it fixes a severe false-positive failure on human-writing-dpo.

Training Procedure

Base model: roberta-base

Key settings:

• Max sequence length: 512
• Batch size: 8
• Evaluation batch size: 64
• Gradient accumulation: 2
• Epochs: 2
• Learning rate: 2e-5
• Weight decay: 0.01
• Warmup ratio: 0.06
• Class weighting: balanced
• Mixed precision: fp16
• Seed: 20260530
• Best checkpoint selection: validation ROC AUC

The selected checkpoint is epoch 1. Epoch 2 increased AI recall slightly but reduced human recall, so epoch 1 was kept.

Evaluation

All-sources held-out test result:

Comparison against the previous mixed-plus-Claude RoBERTa model on the all-sources test set:

Important held-out slice results:

Known Limitations

• The model is sensitive to the distributions represented in the assembled datasets.
• AI text detectors can produce false positives, especially under domain shift.
• The training set is not balanced by source; balanced class loss was used, but source imbalance remains.
• Some evaluation slices are single-label, so they report only human recall or AI recall rather than full binary metrics.
• The model should be calibrated and re-evaluated before use on new domains, languages, or generator families.

Artifacts

This repository contains the full fine-tuned Transformers checkpoint:

• model.safetensors
• config.json
• tokenizer.json
• tokenizer_config.json
• metrics.json
• history.json
• run_config.json
• group_metrics.json