ull-ai-image-detector

fev — AI image detector (ONNX, browser-local)

The model shipped by fev, a Manifest V3 Chrome extension that detects AI-generated images entirely inside the browser — WebGPU where available, WebAssembly everywhere else. No cloud inference, no API, no localhost helper. Nothing about an image ever leaves the device.

fev is short for firmes e verdadeiros — firm and true.

What this is

A fine-tune (v13) of Community Forensics ViT-Small/16@384, exported to ONNX fp32.

The SHA-256 above is pinned in the extension's models.lock.json. Its build verifies these exact bytes and fails if they do not match, so a build either reproduces what was measured or stops.

Measured performance

All figures at the fixed decision rule P(AI) >= 0.65. Balanced accuracy is 0.5 × (AI recall + real recall).

The first row was also measured through the built extension in headless Chrome — 1200/1200 images scored at 0.16 s/image on WebGPU, reproducing the PyTorch figure exactly. The numbers are the extension's, not a notebook's.

Read this before trusting the numbers above

Both sets above draw their real class from COCO / ImageNet, which is also the kind of data the model trained on. The splits are byte- and pHash-disjoint, so this is not leakage — but it is an easy question, and balanced accuracy weights the real class equally with the AI class.

So the real class was re-measured against images from the open web the model has never seen a relative of: 768 Wikimedia Commons files across 7 strata, fetched through Commons' own thumbnailer so they arrive resized and re-encoded the way a CMS delivers them. Its own labels were audited twice (embedded provenance, and Commons category membership); 2 of 770 turned out to be AI and were removed before freezing.

v13 exists because of that table: v11's real class was entirely clean photography, so it had never been penalised for flagging human-made non-photographic content, and it flagged a lot of it. v13 adds 816 Wikimedia hard negatives — art, CGI, paintings, engravings, scans, screenshots.

This is a trade, not a free win. Frontier AI recall fell 0.820 → 0.761 and lexica 0.985 → 0.961; adding negatives moves the model toward "real" everywhere. Net on realistic web content (frontier AI recall + diverse-holdout real recall) it is 0.830 vs v11's 0.818. A v11+v13 ensemble was measured and rejected — mean/max/min fusion all scored below v13 alone, so a second model would cost 2× latency for nothing.

Digital art at 0.636 remains the weakest stratum, and algorithmic fractal renders (Mandelbulb and similar) are near-worst-case for this model.

Usage

The intended consumer is the extension, which handles preprocessing:

git clone https://github.com/angelhd1999/ullExtension && cd ullExtension
cd extension && npm ci && cd ..
node scripts/vendor-ort.mjs
node scripts/fetch-model.mjs      # downloads this file, verifies the SHA-256
node scripts/build-extension.mjs

Then load extension/ unpacked at chrome://extensions.

Direct use with onnxruntime:

import numpy as np, onnxruntime as ort
from PIL import Image

MEAN = np.array([0.48145466, 0.4578275, 0.40821073], np.float32)
STD  = np.array([0.26862954, 0.26130258, 0.27577711], np.float32)

img = Image.open("image.jpg").convert("RGB")
w, h = img.size
s = 440 / min(w, h)
img = img.resize((round(w * s), round(h * s)), Image.BICUBIC)
w, h = img.size
img = img.crop(((w - 384) // 2, (h - 384) // 2, (w - 384) // 2 + 384, (h - 384) // 2 + 384))

x = ((np.asarray(img, np.float32) / 255.0 - MEAN) / STD).transpose(2, 0, 1)[None]
sess = ort.InferenceSession("commfor384_web_v13_fp32.onnx")
logit = sess.run(None, {sess.get_inputs()[0].name: x})[0][0, 0]
p_ai = 1 / (1 + np.exp(-logit))
print(f"P(AI) = {p_ai:.4f} -> {'AI' if p_ai >= 0.65 else 'real'}")

Preprocessing must match exactly. The extension reimplements PIL's fixed-point bicubic resize in JavaScript rather than delegating to drawImage, whose filter is undocumented and varies across Chrome releases — and resampling is part of what this classifier reads.

Limitations

• Real recall drops on non-photographic human-made content (see above); human-made digital art is the weakest stratum at 0.636.
• Frontier AI recall is 0.761 — the cost of the hard-negative training that fixed the real class. v11 is better there (0.820) and worse everywhere else.
• Trained partly on the OpenFake validation split, which is CC BY-NC 4.0. The base weights and this export are MIT; whether a CC BY-NC dataset encumbers the resulting weights is unsettled, and an OpenFake-free variant is tracked in the repository.
• fp16 is not published: in Chrome it falls back to WASM (Dawn does not expose shader-f16) and runs slower than fp32 on WebGPU.

Citation

The base model:

@article{park2024community,
  title={Community Forensics: Using Thousands of Generators to Train Fake Image Detectors},
  author={Park, Jeongsoo and Owens, Andrew},
  journal={arXiv preprint arXiv:2411.04125},
  year={2024}
}