FAIR-EO-CD-benchmark

<div align="center"> <h1>A comprehensive and trustworthy benchmark of AI methods for change detection in Earth observation</h1>

Tadej Tomanič^{1, 2} · Alice Baudhuin¹ · Jan Sotošek¹ · Jure Brence^{1, 3} · Panče Panov^{1, 3} · Nikola Simidjievski^{1, 4} · Dragi Kocev^{1, 3}

<br> ¹Bias Variance Labs, d.o.o.&emsp;&emsp;²University of Ljubljana, Faculty of Mathematics and Physics<br> ³Department of Knowledge Technologies, Jožef Stefan Institute&emsp;&emsp;⁴Télécom Paris, Institut Polytechnique de Paris

<br><br> <a href='https://huggingface.co/bvlabs/FAIR-EO-CD-benchmark'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-FAIR--EO--CD--benchmark-yellow'></a>

</div>

---

Summary

Companion artifacts for the paper, A comprehensive and trustworthy benchmark of AI methods for change detection in Earth observation (currently under review).

This study presents a standardized benchmark for change detection in Earth observation (EO), developed as part of the OSCARS-funded FAIR-EO project. The framework is integrated into the AiTLAS toolbox.

Despite rapid advancements in machine learning for remote sensing, accurately evaluating and comparing change detection models remains a significant challenge. Variations in dataset preprocessing, data splits, and evaluation metrics often lead to inconsistent results, making it difficult to determine whether a new method genuinely outperforms existing ones. To address this, we introduce a comprehensive, transparent, and trustworthy benchmarking framework designed to eliminate these ambiguities. Aligned with the core principles of FAIR (Findable, Accessible, Interoperable, and Reusable) and open science, our framework provides a unified pipeline for the Earth Observation community.

GitHub repo for the benchmark: FAIR-EO-CD-benchmark

GitHub repo for AiTLAS: AiTLAS Toolbox

---

Benchmark Grid

Checkpoints and training curves for a change detection benchmark across 10 datasets, 10 model architectures, and 2 initialisations/training types — 180 runs.

(Note: ChangeFormerV6 and CSSM are released scratch-only; model weights pre-trained on ImageNet-1K were not available, so the grid is 180 runs rather than a full 200).

Logged metrics per run: Loss/train, Loss/val, Accuracy/val, IOU/val, IOU mean/val, and micro/macro/weighted plus per-class (no change, change) Precision, Recall, F1_score.

---

Repository Layout

checkpoints/
  best_checkpoint_<dataset>_<init>_<model>.pth.tar     # 180 files
runs/
  <dataset>_<init>_<model>/
    tensorboard_<dataset>_<init>_<model>.tfevents.0    # 180 files