yolo-rdd2022-benchmark

YOLO × RDD2022 Benchmark

A systematic benchmark of 8 YOLO variants (YOLOv5s through YOLOv12m) on the Road Damage Detection 2022 (RDD2022) dataset, evaluated across 7 country partitions with full multi-seed replication and statistical significance testing.

This repository hosts the trained model weights (best.pt) for all benchmark runs.

---

Benchmark Summary

† YOLOv10s evaluated with its default NMS-free head. Applying standard NMS post-processing recovers +5.43 mAP50 points (0.5694 → 0.6193), making it competitive with YOLOv9s.

Mean mAP50 = average across 7 country partitions. 3-Seed Mean/Std = global val mAP50 across seeds 0, 42, 123 (YOLOv12m and YOLOv12n have seed 0 only).

---

Dataset

RDD2022 covers road damage images from 7 geographic partitions with 4 damage classes:

Country Partition Statistics

---

Results by Country Partition

mAP50

F1 (at Optimal Confidence Threshold)

All optimal confidence thresholds cluster at 0.10–0.20, well below the YOLO default of 0.25. Using default conf=0.25 reduces F1 for most models.

---

Statistical Analysis

Test: Friedman + Nemenyi post-hoc (α = 0.05), N = 7 country partitions, k = 8 YOLO variants.
Critical Difference = 3.97 (Demšar 2006).

Mean Ranks (lower = better)

Significant Pairwise Differences (Nemenyi, p < 0.05)

YOLOv12s and YOLOv12m are statistically indistinguishable from each other and from YOLOv8s/YOLOv9s. Only separation from YOLOv10s and YOLOv12n reaches significance.

---

Seed Replication & Stability

Multi-seed replication (seeds 0, 42, 123) was run for 6 of the 8 models. YOLOv12m and YOLOv12n have seed 0 only.

Maximum inter-seed std is 0.0030 (YOLOv5s). The 15× ratio of architectural spread to max seed variance confirms that model rankings are robust to seed choice for all pairs except YOLOv9s vs YOLOv12s (Δ = 0.0020, ~0.7σ — genuinely ambiguous).

---

Resolution Ablation (YOLOv12s)

800px yields +4.37 mAP50 points over 480px and converges faster (epoch 66 vs 82). All benchmark results use the default 640px unless otherwise noted.

---

Per-Class AP50 — Hardest Partitions (YOLOv12s)

Norway

India

India D10 is completely undetected (only 8 instances in the partition). Norway D40 (pothole) is near-zero despite being a safety-critical class.

---

India Fine-Tuning Experiment

Starting from the global YOLOv12s checkpoint, fine-tuning on a fraction of India-labeled data was tested across 3 seeds and 3 data percentages.

Fine-tuning on 10% India data yields a marginal +0.98 pp gain on India mAP50, but increasing the fine-tune fraction past 10% causes degradation — both on India (overfitting) and on other partitions (catastrophic forgetting).

---

Model Weights

All best.pt files are available in the Files tab. File naming convention:

{model}_seed{seed}_best.pt              # e.g. yolov12s_seed0_best.pt
{model}_seed{seed}_best.pt              # e.g. yolov8s_seed42_best.pt
yolov12s_480px_seed0_best.pt            # resolution ablation
yolov12s_800px_seed0_best.pt            # resolution ablation
india_ft_{pct}pct_seed{seed}_best.pt    # India fine-tune variants

---

Reproduction

Requirements

pip install ultralytics

For YOLOv9, install the WongKinYiu/yolov9 fork. All other models use the ultralytics package.

Dataset Setup

Download RDD2022 from the official source and structure it as:

rdd2022/
├── images/
│   ├── train/
│   └── val/
└── labels/
    ├── train/
    └── val/

Create a rdd2022.yaml data config pointing to your local paths.

Training

from ultralytics import YOLO

model = YOLO("yolov12s.pt")  # pretrained backbone — downloaded automatically by Ultralytics on first run
model.train(
    data="rdd2022.yaml",
    epochs=100,
    imgsz=640,
    seed=0,
    optimizer="auto",
)

Evaluation

model = YOLO("yolov12s_seed0_best.pt")  # load from this repo
metrics = model.val(data="rdd2022.yaml", imgsz=640, conf=0.10)
print(metrics.box.map50)

Use conf=0.10 for optimal F1 (see F1 sweep results). Default conf=0.25 underperforms for this dataset.

YOLOv10s NMS Patch

YOLOv10s uses an NMS-free detection head by default. To reproduce the patched results (+5.43 mAP50):

from ultralytics import YOLO
model = YOLO("yolov10s_seed0_best.pt")
# Enable standard NMS post-processing
metrics = model.val(data="rdd2022.yaml", conf=0.10, iou=0.45, agnostic_nms=False)

---

Key Findings

• YOLOv12s ranks #1 by Friedman mean rank (2.14) but is statistically indistinguishable from YOLOv12m and the YOLOv8s/YOLOv9s cluster.
• Norway is the hardest partition across all models (max mAP50 = 0.2462 by YOLOv12m). Maximum achievable recall on Norway is only 0.6577 even at confidence threshold = 0 — a structural detection ceiling likely driven by extreme class imbalance (77.5% D00) and challenging imaging conditions.
• China (Motorbike) is the easiest partition (all models > 0.88 mAP50). The 70.7 pp spread between hardest and easiest partition is 15.8× the inter-model spread, meaning dataset difficulty dominates architecture choice.
• YOLOv10s significantly underperforms due to its NMS-free head in standard evaluation. With standard NMS applied, it becomes competitive (+5.43 mAP50 points).
• YOLOv12n is the weakest model and the only v12 variant that fails to improve over the v5/v8/v11 generation.
• Optimal confidence for this dataset is 0.10–0.20, not the YOLO default of 0.25.
• India fine-tuning yields diminishing returns: 10% data gives a marginal +1 pp gain; more data causes catastrophic forgetting.

---

Training Configuration

---

Citation

If you use these weights or results, please cite the RDD2022 dataset:

@article{arya2022rdd2022,
  title={RDD2022: A multi-national image dataset for automatic Road Damage Detection},
  author={Arya, Deeksha and Maeda, Hiroya and Ghosh, Sanjay Kumar and Toshniwal, Durga and Mraz, Alexander and Kashiyama, Takehiro and Sekimoto, Yoshihide},
  journal={arXiv preprint arXiv:2209.08538},
  year={2022}
}

---

License

Model weights and benchmark data are released under CC BY 4.0.