AgentGen-Bench-Evaluator

AgentGen-Bench-Evaluator

AgentGen-Bench-Evaluator is the official evaluator for AgentGen-Bench, the benchmark introduced by Search Beyond What Can Be Taught: A Co-Evolution Framework for Search-Enhanced Image Generation. AgentGen-Bench was previously named SearchGen-Bench; the new name reflects its broader use for evaluating agentic generation systems.

The evaluator is trained on top of Qwen3.5-9B. We recommend this released checkpoint for full benchmark reproducibility.

Resources

• Evaluation code and setup
• AgentGen-Bench website and leaderboard
• AgentGen-Bench dataset

Serving with vLLM

Install a recent vLLM release, download serve_evaluator.sh, and run:

pip install vllm
bash serve_evaluator.sh

The endpoint is OpenAI-compatible and serves the model as AgentGen-Bench-Evaluator on port 7070.

Two serving details are required:

• Enforce --chat-template-content-format openai. The evaluator consumes multimodal OpenAI-style message content; relying on automatic format detection can produce incompatible requests.
• For multiple GPUs, use vLLM data parallelism. For example, bash serve_evaluator.sh --gpus 0,1,2,3 starts four data-parallel replicas with tensor parallel size one. The script derives the data-parallel size from the GPU list; --dp and --tp remain available for custom layouts.

An equivalent direct command is:

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve JasperHaozhe/AgentGen-Bench-Evaluator \
  --served-model-name AgentGen-Bench-Evaluator \
  --chat-template-content-format openai \
  --max-model-len 20480 \
  --limit-mm-per-prompt '{"image": 8, "video": 0}' \
  --tensor-parallel-size 1 \
  --data-parallel-size 4 \
  --gpu-memory-utilization 0.75 \
  --port 7070

Use the endpoint with the released evaluation code and setup.

Intended use

This checkpoint is intended to reproduce AgentGen-Bench evaluation of generated images across knowledge and rendering dimensions. Like any learned judge, it may inherit biases from its training data and base model. Scores should be interpreted using the released protocol and aggregated across the benchmark, not as definitive judgments of individual images.

License

Apache-2.0. Users must also comply with the terms of the Qwen3.5-9B base model.