ai-illust-detector-siglip2-so400m-patch16-384

AI-vs-Human Illustration Classifier (SigLIP2 so400m)

This is a binary classifier that predicts whether an illustration is AI-generated (ai) or human-made (human). It finetunes the vision encoder of google/siglip2-so400m-patch16-384 with a 2-class linear head. The text tower is dropped.

• id2label = {0: "human", 1: "ai"}
• Output: 2 logits → P(ai) = softmax(logits)[1].

Decision threshold

This model is not used at the naive 0.5 cut. The probabilities are not calibrated to 0.5. the operating point was tuned on the validation set to maximize balanced accuracy:

decide "ai"  iff  P(ai) >= 0.0368        # exact value in metrics.json["best_threshold"]

The Hugging Face pipeline / argmax uses 0.5, which is more conservative about predicting ai. For the reported numbers, apply the tuned threshold above (or retune one on your own data). Both operating points are strong (see metrics).

Usage

This is a standard transformers image classifier, so it loads with AutoModelForImageClassification / AutoImageProcessor. The only specific detail is the decision threshold (see above): the tuned operating point lives in the bundled metrics.json as best_threshold, not the default 0.5.

import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import AutoModelForImageClassification, AutoImageProcessor

repo = "Minachist/ai-illust-detector-siglip2-so400m-patch16-384"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModelForImageClassification.from_pretrained(repo).to(device).eval()
processor = AutoImageProcessor.from_pretrained(repo)

# Tuned operating point from metrics.json (falls back to 0.0368 if absent).
metrics = json.load(open(hf_hub_download(repo, "metrics.json")))
THRESHOLD = float(metrics.get("best_threshold", 0.0368))

img = Image.open("example.png").convert("RGB")
inputs = processor(images=img, return_tensors="pt").to(device)
with torch.no_grad():
    p_ai = model(**inputs).logits.softmax(-1)[0, 1].item()

label = "ai" if p_ai >= THRESHOLD else "human"
print(f"P(ai)={p_ai:.4f} -> {label}")

Note: the Hugging Face pipeline / argmax decides at 0.5; to reproduce the reported numbers, apply THRESHOLD from metrics.json as shown above.

Preprocessing

Images of any resolution / aspect ratio are squashed (aspect ratio ignored) to 384×384 with bicubic resampling, rescaled to [0,1], then normalized with mean = std = 0.5 per channel. Identical between training and inference. The bundled AutoImageProcessor already encodes this.

Validation metrics

Evaluated on a held-out, shortcut-controlled set (robust variant), n = 9,988 (human 6,424 / ai 3,564). Checkpoint step 12000.

At the operating point P(ai) >= 0.0368:

Threshold-free / argmax:

The ~0 gap between the raw, robust, and adversarial variants indicates the detector is not leaning on format/resolution shortcuts (see Training).

Evaluation variants (raw / robust / adversarial)

Custom evaluation variants for this project. Since the training data might contain format and resolution biases that are irrelevant to the content, the identical images are rendered in three ways for evaluation to isolate whether the model is looking at the illustration content or relying on them.

If the classifier depends on shortcuts, accuracy would drop on the robust or adversarial. In practice, the AUROC of the 3 variants is nearly identical (see metrics.json). The model judges based on image content, not spurious correlations of format or resolution.

Training

The model uses the vision encoder of google/siglip2-so400m-patch16-384 and adds a classification head. The architecture drops the text tower.

The dataloader samples the classes at a 1:1 ratio. A 2% holdout split per image is excluded from training.

The loss function is cross entropy without label smoothing. The evaluation pipeline calculates a threshold for the decision, so the system outputs probabilities without calibration.

The optimizer is AdamW. The learning rate is 1.5e-5 for the backbone and 1.0e-4 for the head. Weight decay is 0.05 and gradient clipping is 1.0. The learning rate schedule is cosine with 500 warmup steps and a cap of 16000 steps. The batch size is 256 (16 per GPU, 2 GPUs, 8 gradient accumulation steps). Training runs in bf16 precision with TF32 enabled, utilizing DistributedDataParallel across 2 GPUs.

Data augmentation applies to all inputs. Operations include x-axis flips with probability 0.5. JPEG recompression applies with probability 0.9, utilizing quality values between 40 and 95, with a 25% probability of a repetition, chroma-subsampling, and grid-phase jitter. Resolution normalization scales width or height to a value between 384 and 1152 with probability 0.9 before scaling to 384. The operations suppress learning of format shortcuts.

Checkpoint selection evaluates accuracy per class using the threshold from calculation. AUROC reaches 0.999 and cross-entropy increases while accuracy improves, so neither metric serves for selection. Training stopped at step 12000 and the release uses the checkpoint from step 12000.

Intended use & limitations

This is a research artifact. Behavior on other domains is out of scope. Generalization to generative models not represented in the training data is not guaranteed. New generators, post-processing, or heavy recompression can and will degrade accuracy. The reported metrics come from one in-distribution validation set and will be optimistic for data in the wild. Do not use it as a sole source of truth for high-stakes or punitive decisions.

Disclaimer

This model is released under the Apache-2.0 license. You are free to use it as you see fit, but it is provided "AS IS", WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. The developer(s) assume absolutely no liability for any direct, indirect, incidental, or consequential damages arising out of the use, or inability to use, this model and software.

I strongly advise against the following high-risk use cases. If you choose to engage in them, you do so entirely at your own risk and bear sole responsibility for any consequences:

• Harassment or witchhunting: Using this tool to publicly target, harass, or discredit artists or creators by accusing them of using AI.
• Deception: Using this model to bypass or exploit other detection systems, or to train adversarial attacks against other models.
• Legal evidence: Presenting the model's probabilistic predictions as definitive legal proof in copyright disputes or other legal proceedings.

The model is probabilistic and known to make mistakes (false positives and false negatives). Any decisions made based on its output are solely the responsibility of the user.

Citation

This model is an unofficial fine-tune. All credit for the base SigLIP 2 architecture and pre-trained weights goes to the original authors at Google. If you use this model or build upon it, please consider citing their paper:

@misc{tschannen2025siglip2multilingualvisionlanguage,
      title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features}, 
      author={Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier Hénaff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},
      year={2025},
      eprint={2502.14786},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.14786}, 
}