imageeval2026-task2-craibench

ImageEval 2026 · Task 2 — CRAI-Bench

System artefacts for CRAI-Bench: given a reference image of a Qatari cultural scene, the caption used to prompt a text-to-image model, and the generated image, predict five cultural accuracy dimensions.

CRAI_composite = 0.30*CEA + 0.20*CC + 0.20*CS + 0.20*CI - 0.10*HP

Ranking metric is Spearman on the composite; MAE is the tiebreaker.

Results — dev (n=40), fit on train, held out

Per-dimension Spearman for the submitted system: CEA 0.716, CC 0.711, CS 0.726, CI 0.730, HP -0.095.

Method

A VLM judge scores the five dimensions from (reference image, caption, generated image); a per-dimension ridge maps [5 raw dims + caption-version one-hot] to the human scores, fit on the 120 train instances. The composite is always recomputed with the official formula, never predicted.

Two facts drive this:

1. The organisers note GPT-4 "systematically overestimates cultural accuracy". It does, on every dimension — bias +0.142 (CEA) to +0.247 (CI), +0.182 on the composite.
2. Spearman is rank-based, so monotone recalibration cannot change it. Calibration buys MAE only; improving the ranking needs new signal. Caption version supplies it.

Caveats stated plainly

• n = 40. Spearman's standard error is roughly ±0.16. Our 0.7064 versus the version-only control's 0.6823 is not a significant gap.
• Train CV was optimistic: 0.8159 under 6-fold grouped CV versus 0.7064 on dev.
• A model that never sees an image scores 0.6823. Caption version is close to a label — a benchmark design artefact that any honest system paper must report.
• The organisers' GPT-4 baseline did not reproduce: their shipped gold_llm.tsv scores 0.2153 on dev, not the reported 0.6250.
• Humans almost never apply a hallucination penalty (mean HP 0.008 vs GPT-4's 0.173), and GPT-4's HP ranking is anti-correlated with theirs (rho -0.114).

Data and licence

Built on the organisers' CRAI-Bench release, distributed via their Google Drive and subject to their terms; the task repository asks that data not be committed publicly. This repo contains no dataset content — no images, no captions, no gold labels. Contact: [email protected]