dactyl-ai-text-detector

<div align="center"> <img src="https://raw.githubusercontent.com/ShantanuT01/dactyl_generation/refs/heads/main/docs/assets/icons8-pterodactyl-48.svg" alt="DACTYL logo" width="64"/> <h1> DACTYL — AI-Generated Text Detector</h1> </div> <div align="center">

</div>

DACTYL is a microsoft/deberta-v3-large classifier fine-tuned for AI-generated text (AIG) detection using Empirical X-Risk Minimization (EXM). It is trained on the DACTYL-train dataset — a challenging corpus of one-shot/few-shot and continued-pre-trained (CPT) model generations — and evaluated on a held-out test set containing unseen models not present during training, demonstrating robust out-of-distribution generalization.

Why EXM?

Standard binary cross-entropy (BCE) (or empirical risk minimization, ERM) training optimizes average performance, which can mask poor behavior at the low false-positive rates that matter most in real deployments. EXM instead directly optimizes X-risk metrics (such as partial AUROC) that emphasize performance at these critical operating points. As reported in the accompanying paper, while ERM-trained classifiers marginally outperform EXM classifiers on the DACTYL test set, EXM classifiers generalize significantly better to out-of-distribution data — by as much as 50 macro-F1 points in a student essay detection scenario.

Benchmark Results

DACTYL was benchmarked against 4 other open-source AI-generated text detectors across 9 datasets. Evaluation metrics are AUROC (threshold-independent) and F1 at a default threshold of 0.5.

The DACTYL test set is particularly challenging: it contains one-shot/few-shot generations (where LLMs are prompted with human text as an example) and outputs from domain-specific CPT models — generation conditions that most existing detectors were not designed for and struggle with.

Note on external datasets: The 8 external benchmark datasets consist predominantly of zero-shot generations, which are stylistically further from human writing than the one-shot/few-shot generations DACTYL was trained on. This represents a distribution shift in the opposite direction — DACTYL is optimized for harder, more human-like AI text, and these datasets are comparatively easier. DACTYL's competitive performance on them despite this mismatch is encouraging, but scores on the external datasets should not be taken as the primary measure of its intended capability.

<div align="center"> <svg xmlns="http://www.w3.org/2000/svg" width="800" height="320" style="background:#12121a;font-family:'IBM Plex Mono',monospace;border-radius:12px"> <line x1="60" y1="260.0" x2="670" y2="260.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="264.0" text-anchor="end" fill="#6a6a8a" font-size="10">0.5</text> <line x1="60" y1="216.0" x2="670" y2="216.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="220.0" text-anchor="end" fill="#6a6a8a" font-size="10">0.6</text> <line x1="60" y1="172.0" x2="670" y2="172.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="176.0" text-anchor="end" fill="#6a6a8a" font-size="10">0.7</text> <line x1="60" y1="128.0" x2="670" y2="128.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="132.0" text-anchor="end" fill="#6a6a8a" font-size="10">0.8</text> <line x1="60" y1="84.0" x2="670" y2="84.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="88.0" text-anchor="end" fill="#6a6a8a" font-size="10">0.9</text> <line x1="60" y1="40.0" x2="670" y2="40.0" stroke="#2a2a3d" stroke-width="1"/> <text x="52" y="44.0" text-anchor="end" fill="#6a6a8a" font-size="10">1.0</text> <rect x="69.8" y="82.2" width="39.0" height="177.8" fill="#7c6af7" opacity="0.7" rx="2"/> <text x="89.3" y="78.2" text-anchor="middle" fill="#e8e8f0" font-size="9">0.904</text> <rect x="118.6" y="154.8" width="39.0" height="105.2" fill="#4ecca3" opacity="0.7" rx="2"/> <text x="138.1" y="150.8" text-anchor="middle" fill="#e8e8f0" font-size="9">0.739</text> <text x="113.7" y="278" text-anchor="middle" fill="#6a6a8a" font-size="11" font-weight="normal">Desklib</text> <rect x="191.8" y="83.1" width="39.0" height="176.9" fill="#7c6af7" opacity="0.7" rx="2"/> <text x="211.3" y="79.1" text-anchor="middle" fill="#e8e8f0" font-size="9">0.902</text> <rect x="240.6" y="147.8" width="39.0" height="112.2" fill="#4ecca3" opacity="0.7" rx="2"/> <text x="260.1" y="143.8" text-anchor="middle" fill="#e8e8f0" font-size="9">0.755</text> <text x="235.7" y="278" text-anchor="middle" fill="#6a6a8a" font-size="11" font-weight="normal">Fakespot</text> <rect x="313.8" y="84.0" width="39.0" height="176.0" fill="#7c6af7" opacity="1" rx="2" stroke="#fff" stroke-width="1.5"/> <text x="333.3" y="80.0" text-anchor="middle" fill="#e8e8f0" font-size="9">0.900</text> <rect x="362.6" y="140.3" width="39.0" height="119.7" fill="#4ecca3" opacity="1" rx="2" stroke="#fff" stroke-width="1.5"/> <text x="382.1" y="136.3" text-anchor="middle" fill="#e8e8f0" font-size="9">0.772</text> <text x="357.7" y="278" text-anchor="middle" fill="#e8e8f0" font-size="11" font-weight="bold">DACTYL</text> <rect x="435.8" y="97.6" width="39.0" height="162.4" fill="#7c6af7" opacity="0.7" rx="2"/> <text x="455.3" y="93.6" text-anchor="middle" fill="#e8e8f0" font-size="9">0.869</text> <rect x="484.6" y="169.8" width="39.0" height="90.2" fill="#4ecca3" opacity="0.7" rx="2"/> <text x="504.1" y="165.8" text-anchor="middle" fill="#e8e8f0" font-size="9">0.705</text> <text x="479.7" y="278" text-anchor="middle" fill="#6a6a8a" font-size="11" font-weight="normal">SuperAnnotate</text> <rect x="557.8" y="106.0" width="39.0" height="154.0" fill="#7c6af7" opacity="0.7" rx="2"/> <text x="577.3" y="102.0" text-anchor="middle" fill="#e8e8f0" font-size="9">0.850</text> <rect x="606.6" y="230.5" width="39.0" height="29.5" fill="#4ecca3" opacity="0.7" rx="2"/> <text x="626.1" y="226.5" text-anchor="middle" fill="#e8e8f0" font-size="9">0.567</text> <text x="601.7" y="278" text-anchor="middle" fill="#6a6a8a" font-size="11" font-weight="normal">Oxidane</text> <line x1="60" y1="40" x2="60" y2="260" stroke="#2a2a3d" stroke-width="1"/> <rect x="510" y="48" width="12" height="12" fill="#7c6af7" rx="2"/> <text x="526" y="58" fill="#e8e8f0" font-size="10">AUROC</text> <rect x="580" y="48" width="12" height="12" fill="#4ecca3" rx="2"/> <text x="596" y="58" fill="#e8e8f0" font-size="10">F1</text> <text x="60" y="28" fill="#e8e8f0" font-size="12" font-weight="bold">Mean AUROC &amp; F1 Score by Classifier</text> </svg> </div>

AUROC

F1 (threshold = 0.5)

DACTYL ranks 1st in mean F1 and 3rd in mean AUROC among the five evaluated detectors.

Note: The APT-Eval dataset yields low F1 scores across all classifiers at the default threshold of 0.5. The relatively higher AUROC (~0.80) on this dataset suggests the classifier retains discriminative ability but may benefit from threshold calibration for this domain.

Training Data

DACTYL is trained on the DACTYL-train dataset, which focuses on two underexplored generation conditions:

• One-shot/few-shot generations — AI text produced when the model is given a human-written example, making it stylistically closer to human writing and harder to detect
• CPT model generations — text from domain-specific continued-pre-trained small language models, fine-tuned with memory-efficient full-parameter optimization

Training Configuration

We provide full training details for this model.

{
    "training_split": "training",
    "evaluation_split": "testing",
    "num_epochs": 1,
    "model_path": "microsoft/deberta-v3-large",
    "tokenizer": "microsoft/deberta-v3-large",
    "optimizer": "SOTAs",
    "optimizer_type": "libauc",
    "optimizer_args": {
        "lr": 1e-05
    },
    "loss_fn": "tpAUC_KL_Loss",
    "reset_classification_head": false,
    "loss_type": "libauc",
    "loss_fn_args": {
        "data_len": 466005
    },
    "needs_loss_fn_as_parameter": false,
    "training_args": {
        "batch_size": 16,
        "needs_sampler": true,
        "needs_index": true,
        "shuffle": false,
        "sampling_rate": 0.5,
        "apply_sigmoid": true
    },
}

Intended Use

• Detecting AI-generated text in research and content moderation contexts
• Benchmarking against other open-source detection systems

Limitations

• Zero-shot distribution shift: Most existing AIG detection datasets use zero-shot generations, whereas DACTYL is trained on harder one-shot/few-shot and CPT-generated text. This mismatch means DACTYL-trained classifiers may slightly underperform on zero-shot-heavy benchmarks relative to detectors trained directly on those distributions — the external benchmark scores above should be interpreted with this in mind.
• EXM vs ERM tradeoff: EXM classifiers generalize better out-of-distribution but may underperform ERM classifiers on the DACTYL test set itself. If your deployment domain closely matches DACTYL-train, an ERM variant may yield higher in-distribution F1.
• CPT model degradation: As noted in the paper, continued pre-trained models are harder to detect even for DACTYL-trained classifiers — performance degrades relative to standard LLM outputs.
• Threshold sensitivity: F1 scores at the default threshold of 0.5 vary significantly across domains. Users are encouraged to calibrate the threshold for their specific use case using a small labeled sample.
• Like all detectors, DACTYL may degrade against future unseen model architectures.

Citation

@misc{thorat2025dactyl,
  title         = {DACTYL: Diverse Adversarial Corpus of Texts Yielded from Large Language Models},
  author        = {Shantanu Thorat and Andrew Caines},
  year          = {2025},
  eprint        = {2508.00619},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2508.00619}
}