ai-illust-detector-siglip2-so400m-patch16-naflex

AI-vs-Human Illustration Classifier (SigLIP2 so400m NaFlex)

This is a binary classifier that predicts whether an illustration is AI-generated (ai) or human-made (human). It finetunes the vision encoder of google/siglip2-so400m-patch16-naflex with a 2-class linear head. The text tower is dropped.

Unlike the fixed-resolution SigLIP2 checkpoints, the NaFlex variant preserves the native aspect ratio and processes a variable patch budget (here max_num_patches = 1024, ≈512px-equivalent), so the model sees more detail and the geometry that often carries the tell (proportion errors, malformed shapes) is not distorted by a square squash.

• id2label = {0: "human", 1: "ai"}
• Output: 2 logits → P(ai) = softmax(logits)[1].

Decision threshold

This model is not used at the naive 0.5 cut. The probabilities are not calibrated to 0.5. the operating point was tuned on the validation set to maximize balanced accuracy:

decide "ai"  iff  P(ai) >= 0.9151        # exact value in metrics.json["best_threshold"]

The Hugging Face pipeline / argmax decides at 0.5, which predicts ai more readily than this tuned point (it trades a little human-precision for ai-recall). For the reported numbers, apply the tuned threshold above (or retune one on your own data). Both operating points are strong (see metrics).

Usage

This is a standard transformers image classifier, so it loads with AutoModelForImageClassification / AutoImageProcessor. The only specific detail is the decision threshold (see above): the tuned operating point lives in the bundled metrics.json as best_threshold, not the default 0.5.

import json
import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from transformers import AutoModelForImageClassification, AutoImageProcessor

repo = "Minachist/ai-illust-detector-siglip2-so400m-patch16-naflex"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = AutoModelForImageClassification.from_pretrained(repo).to(device).eval()
processor = AutoImageProcessor.from_pretrained(repo)

# Tuned operating point from metrics.json (falls back to 0.9151 if absent).
metrics = json.load(open(hf_hub_download(repo, "metrics.json")))
THRESHOLD = float(metrics.get("best_threshold", 0.9151))

img = Image.open("example.png").convert("RGB")
inputs = processor(images=img, return_tensors="pt").to(device)
with torch.no_grad():
    p_ai = model(**inputs).logits.softmax(-1)[0, 1].item()

label = "ai" if p_ai >= THRESHOLD else "human"
print(f"P(ai)={p_ai:.4f} -> {label}")

Note: the Hugging Face pipeline / argmax decides at 0.5; to reproduce the reported numbers, apply THRESHOLD from metrics.json as shown above. The AutoImageProcessor returns the NaFlex dict (pixel_values, pixel_attention_mask, spatial_shapes); model(**inputs) consumes it directly.

Preprocessing

Unlike the fixed-resolution SigLIP2 models, this NaFlex checkpoint does not squash images to a fixed square. Each image is resized, preserving its native aspect ratio so it tiles into at most max_num_patches = 1024 patches of 16×16 px, rescaled to [0,1], then normalized with mean = std = 0.5 per channel. The 16×16 patches are flattened to length-768 vectors and the sequence is padded to exactly 1024, with a pixel_attention_mask marking the real patches and spatial_shapes recording each image's patch grid. So the model input is a dict — {pixel_values (B,1024,768), pixel_attention_mask (B,1024), spatial_shapes (B,2)} — not a (B,3,H,W) tensor. The deterministic resize/normalize is identical between training and inference. The bundled AutoImageProcessor (a Siglip2ImageProcessor) already encodes this, and its max_num_patches sets the inference resolution.

Validation metrics

Evaluated on a held-out, shortcut-controlled set (robust variant), n = 9,988 (human 6,451 / ai 3,537). Checkpoint step 3000.

At the operating point P(ai) >= 0.9151:

Threshold-free / argmax:

The ~0 gap between the raw, robust, and adversarial variants indicates the detector is not leaning on format/resolution shortcuts (see Training).

Evaluation variants (raw / robust / adversarial)

Custom evaluation variants for this project. Since the training data might contain format and resolution biases that are irrelevant to the content, the identical images are rendered in three ways for evaluation to isolate whether the model is looking at the illustration content or relying on them.

If the classifier depends on shortcuts, accuracy would drop on the robust or adversarial. In practice, the AUROC of the 3 variants is nearly identical (see metrics.json). The model judges based on image content, not spurious correlations of format or resolution.

Training

The model uses the vision encoder of google/siglip2-so400m-patch16-naflex and adds a classification head. The architecture drops the text tower. This is the Siglip2 (NaFlex) architecture: it preserves the native aspect ratio and consumes a patchified, variable-length sequence (padded to max_num_patches = 1024) rather than a fixed 384×384 square.

The dataloader samples the classes at a 1:1 ratio. A 2% holdout split per image is excluded from training.

The loss function is cross entropy without label smoothing. The evaluation pipeline calculates a threshold for the decision, so the system outputs probabilities without calibration.

The optimizer is AdamW. The learning rate is 1.5e-5 for the backbone and 1.0e-4 for the head. Weight decay is 0.05 and gradient clipping is 1.0. The learning rate schedule is cosine with 500 warmup steps and a cap of 16000 steps. The batch size is 256 (8 per GPU, 2 GPUs, 16 gradient accumulation steps). Training runs in bf16 precision with TF32 enabled, utilizing DistributedDataParallel across 2 GPUs. The optimizer, schedule, augmentation, and selection knobs are kept identical to the fixed-resolution so400m run, so this isolates the effect of resolution and aspect ratio rather than a retuned recipe.

Data augmentation applies to all inputs. Operations include x-axis flips with probability 0.5. JPEG recompression applies with probability 0.9, utilizing quality values between 40 and 95, with a 25% probability of a repetition, chroma-subsampling, and grid-phase jitter. Resolution normalization scales width or height to a value between 384 and 1152 with probability 0.9 before the processor fits the image to the patch budget. Even without a fixed square, "how much an image was downscaled to fit the budget" is a downscale-signature shortcut, so randomizing it class-agnostically suppresses learning of format shortcuts.

Checkpoint selection monitors balanced accuracy at the tuned threshold on the robust variant — the metric the inference pipeline actually ships. AUROC saturates near 0.999 and ranks late checkpoints poorly, and cross-entropy drifts up with rising confidence even while accuracy improves, so neither serves for selection. Early stopping terminates the run when balanced accuracy stops improving; the release uses the best checkpoint, from step 3000.

Intended use & limitations

This is a research artifact. Behavior on other domains is out of scope. Generalization to generative models not represented in the training data is not guaranteed. New generators, post-processing, or heavy recompression can and will degrade accuracy. The reported metrics come from one in-distribution validation set and will be optimistic for data in the wild. Do not use it as a sole source of truth for high-stakes or punitive decisions.

Disclaimer

This model is released under the Apache-2.0 license. You are free to use it as you see fit, but it is provided "AS IS", WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. The developer(s) assume absolutely no liability for any direct, indirect, incidental, or consequential damages arising out of the use, or inability to use, this model and software.

I strongly advise against the following high-risk use cases. If you choose to engage in them, you do so entirely at your own risk and bear sole responsibility for any consequences:

• Harassment or witchhunting: Using this tool to publicly target, harass, or discredit artists or creators by accusing them of using AI.
• Deception: Using this model to bypass or exploit other detection systems, or to train adversarial attacks against other models.
• Legal evidence: Presenting the model's probabilistic predictions as definitive legal proof in copyright disputes or other legal proceedings.

The model is probabilistic and known to make mistakes (false positives and false negatives). Any decisions made based on its output are solely the responsibility of the user.

Citation

This model is an unofficial fine-tune. All credit for the base SigLIP 2 architecture and pre-trained weights goes to the original authors at Google. If you use this model or build upon it, please consider citing their paper:

@misc{tschannen2025siglip2multilingualvisionlanguage,
      title={SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features}, 
      author={Michael Tschannen and Alexey Gritsenko and Xiao Wang and Muhammad Ferjad Naeem and Ibrahim Alabdulmohsin and Nikhil Parthasarathy and Talfan Evans and Lucas Beyer and Ye Xia and Basil Mustafa and Olivier Hénaff and Jeremiah Harmsen and Andreas Steiner and Xiaohua Zhai},
      year={2025},
      eprint={2502.14786},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2502.14786}, 
}