aion-model-clip-linear-probe-v1

AI or Not CLIP linear probe, results card only

This repository is a citable results card for the face-only CLIP linear-probe experiment. It contains no model weights, checkpoint, executable scorer, feature cache, or image data. The model files are deliberately not published.

Main result, and why it is not a deployment claim

The leak-free leave-one-generator-out probe's worst-fold AUC is 0.9222 on the OpenRL/DeepFakeFace face benchmark. Effective K is about 2 across the three folds because Stable Diffusion and Stable Diffusion Inpainting share a base model, so the worst fold is the headline number. The mean is not the headline.

On the same benchmark, the one-bit source-file feature is_square_512 scores 1.0000 AUC. It beats the probe without looking at image pixels. That is the main limitation of this result, not a footnote. Every AI image in the source is 512 by 512 while the real pool is variable-sized.

JPEG q50 cuts held-out AUC by 0.15 to 0.27, leaving 0.68 to 0.83. The worst measured cell in the negative-source-swap matrix is 0.4771 AUC, below chance, for the leak-free probe on Open Images face crops after JPEG q50.

What was invalidated

Two earlier runs were invalidated by a held-out-family leak. Their nominally held-out family entered training, so their reported held-out score is in-distribution evidence, not cross-generator evidence. They are kept in the source record for auditability and are not evidence for this detector.

Calibration

Calibration is poor on the worst fold: ECE is 0.1057 for InsightFace. Treat scores as ranking outputs from this narrow experiment, not calibrated probabilities and not confidence that an image is AI-generated.

Training data and release gate

The probe was trained on faces only from OpenRL/DeepFakeFace. It is out of distribution on landscapes, objects, documents, screenshots, artwork, video frames, and anything else that is not a face.

The source card's parsed license is openrail, while prose in that same card includes an Apache-2.0 block. The project's assert_permissive_release() gate refuses a permissive weight release because the source terms contradict each other. The general-imagery refuse_weight_export() gate also refuses probes fitted on Synthbuster or RAISE-1k because their terms are non-commercial. No bypass was used.

This Hub repository declares license: other because it has no released model artifact and does not grant a model-weight license. The text of this results card is available under CC BY 4.0. That does not grant rights to the withheld weights or to upstream data.

Intended use

Use this as a research record about benchmark shortcuts, cross-generator evaluation, and compression sensitivity. It can help someone decide what to audit before using a benchmark.

Do not use this for accusations

Do not use this output to accuse a person of faking, fabricating, cheating, or lying about an image. Do not use it as evidence of authorship or in academic misconduct, employment, hiring, immigration, legal, journalistic-sourcing, or automated moderation decisions. A score from this faces-only, shortcut-contaminated experiment is not proof about a person.

At low prevalence, false positives dominate flags even at a low false-positive rate. docs/ETHICS.md includes a worked 1% prevalence example where a detector with 95% true-positive rate and 1% false-positive rate produces about 990 human-image flags among about 1,940 flags. A single flag is not a verdict.

Sources and reproduction

The source repository is tobilicous/ai-or-not. See docs/RESULTS.md, docs/SHORTCUT_ANALYSIS.md, docs/ETHICS.md, artifacts/negswap/metrics.json, and the aion.hub license gates. The source checkout recorded for this card is 09ee04683292af590e53a51acbd177675a83d649.