Imag-Eval

IMAG-EVAL

<p align="center"> <b>IMAG-EVAL:</b> A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation </p>

---

🚀 Overview

IMAG-EVAL is a controlled benchmark designed to evaluate instruction-following capabilities in Text-to-Image (T2I) generation models.

Unlike prior benchmarks that primarily rely on prompt length or isolated skill evaluation, IMAG-EVAL explicitly disentangles linguistic complexity from compositional complexity by independently varying:

• the number of grounded object instances,
• the number of compositional constraints,
• and the combination of evaluated skills.

The benchmark provides interpretable diagnostic evaluations and enables fine-grained analysis of failure modes in modern Text-to-Image systems.

---

📊 Benchmark Statistics

Evaluated Skills

• 🔢 Counting: Generating the exact number of requested object instances.
• 🎨 Color Attribution: Correctly assigning the specified color to each relevant instance.
• 📏 Size Relations: Preserving the required relative size constraints between instances (e.g., larger than, smaller than).
• 📍 Spatial Relations: Correctly grounding spatial relationships between instances (e.g., A is to the right of B).
• 😊 Emotion Attribution: Accurately rendering the specified facial expressions and emotions.
• 🔤 Text Rendering: Faithfully reproducing the target text, including prompts containing spelling variations or typographical errors.
• 🧩 Cohesiveness: Generating anatomically plausible, structurally consistent, and visually coherent instances and scenes.

---

📂 Repository Structure

prompts/

Contains the benchmark prompt definitions used during evaluation.

Each JSON file includes:

• prompt metadata,
• skill combinations,
• difficulty levels,
• scene specifications,
• synthetic prompts,
• evaluation constraints.

skill_codes.csv

Mapping between benchmark identifiers and skill combinations.

Example:

These codes are used throughout the benchmark generation and evaluation pipeline.

texts.csv

Reference textual annotations used for Text Rendering evaluation.

Each row contains:

• prompt identifier,
• expected textual content,
• corresponding benchmark reference.

This file is used to compute Word Error Rate (WER) during evaluation.

---

🎯 Supported Tasks

IMAG-EVAL is intended for:

• Text-to-Image Generation
• Instruction Following
• Compositional Reasoning
• Multimodal Grounding
• Text Rendering Evaluation
• Benchmarking and Diagnostic Analysis

---

📈 Evaluation Protocol

Models are evaluated on seven dimensions:

---

⚠️ Limitations

• The benchmark relies on COCO object categories.
• Emotion attribution is currently restricted to human instances.
• Prompts are written in English.

---

🤗 Related Resources

• 📄 Paper [EMNLP 2026]: IMAG-EVAL: A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation
• 💻 Code: https://github.com/Justsecret123/Imag-Eval

---

📚 Citation (pending EMNLP 2026 proceedings)

@misc{serouis2026imageval,
  title={IMAG-EVAL: A Language-Grounded Framework for Interpretable Text-to-Image Instruction Following Evaluation},
  author={Serouis, Ibrahim Mohamed and Jaramillo Duque, David},
  year={2026},
  note={Accepted at the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)}
}

---

📜 License

Please refer to the repository license for terms of use.