icl_best50

Cross-Image ICL Best-50 Eval Package

Test set for cross-image in-context-learning object detection. 8 classes × 50 query images per class = 400 samples per K, K ∈ {1, 2, 4, 8}.

Layout

cross_icl_best50_package/
├── README.md                  (this file)
├── inputs/
│   ├── inputs_k1.jsonl        (400 samples, K=1)
│   ├── inputs_k2.jsonl        (400 samples, K=2)
│   ├── inputs_k4.jsonl        (400 samples, K=4)
│   └── inputs_k8.jsonl        (400 samples, K=8)
├── support_images/
│   ├── K1/<class>/<stem>__K1.jpg   (red boxes drawn, K=1 subset)
│   ├── K2/<class>/<stem>__K2.jpg
│   ├── K4/<class>/<stem>__K4.jpg
│   └── K8/<class>/<stem>__K8.jpg
└── query_images/
    └── <class>/<file>              (clean, no annotations)

All image paths inside inputs/*.jsonl are package-relative. After untar, they resolve correctly from the package root.

Sample schema (one JSONL line)

The 2 prompt templates (randomly chosen per sample):

A: "Please detect all objects belonging to the same category as the boxes [{boxes}] in the second image. The boxes are marked in the first image."
B: "The first image shows reference objects marked by red boxes at [{boxes}]. Please detect all objects of the same category in the second image."

How to run inference on another model

For each sample, build a multi-modal user message:

[
  {"type": "image", "path": support_image},
  {"type": "image", "path": query_image},
  {"type": "text",  "text": prompt},
]

Apply your model's chat template, then run inference.

Parsing model output

Boxes are emitted as <bbox_start><xA><yB><xC><yD><bbox_end> with A,B,C,D ∈ [0, 1024]. Extract with regex:

import re
TOK = re.compile(r"<x(\d+)><y(\d+)><x(\d+)><y(\d+)>")
def parse(text: str) -> list[list[int]]:
    boxes = []
    for m in TOK.finditer(text or ""):
        x1, y1, x2, y2 = (int(v) for v in m.groups())
        if 0 <= x1 <= 1024 and 0 <= y1 <= 1024 and x2 > x1 and y2 > y1:
            boxes.append([x1, y1, x2, y2])
    return boxes

Since all images are 1024×1024, the grid integers are also pixel coordinates. For non-1024 images: pixel = grid / 1024 * img_size.

Evaluation metrics

Recommended (all per-class micro, then average over classes if you want a single number):

• P@50 / R@50 / F1@50 at IoU 0.50 (greedy match)
• mF1@50:95 = mean over t ∈ {0.50, 0.55, ..., 0.95} of F1 at t (COCO-style IoU sweep, score-free)
• mIoU@50:95 = mean over t of (Σ IoU of TP pairs at t / num total GT)
• mIoU_best = mean over GT boxes of max IoU with any pred (threshold-free)

Reference evaluation scripts (from the original Qwen run):

• _remote_best50_mf1_5095.py — P/R/F1/mF1
• _remote_best50_miou_5095.py — mIoU@50:95 + mIoU_best
• _remote_best50_per_class_k8_prf_miou.py — per-class table

To use them with your model's output, write a JSONL with at least these fields per sample:

{"class": ..., "k": ..., "query_gt_boxes_pixel": [...], "pred_boxes_pixel": [...]}

and point the script's BEST50 / pred_k{k}.jsonl to your file.

Reference metrics (Qwen2.5-VL, ckpt_0514_stage2_iter12629)

clean7 (excl. square_cage), micro, K=8:

• P@50 = 0.6702
• R@50 = 0.8240
• F1@50 = 0.7392
• mF1@50:95 = 0.4113
• mIoU@50:95 = 0.3729
• mIoU_best = 0.6365

Per-class @ K=8 (see metrics_per_class_k8_prf_miou.json for full breakdown).