UrbanOmniDetect

<p align="center"> <h1 align="center">UrbanOmniDetect</h1> <h3 align="center">Calibration-Free View-Agnostic Monocular 3D Object Detection for Urban Scenes</h3> <p align="center"> <a href="https://scholar.google.com/citations?user=306TgWoAAAAJ">Mehmet Kerem Turkcan</a>&emsp; <a href="https://scholar.google.com/citations?user=GP7T1fgAAAAJ">Devika Gumaste</a>&emsp; <a href="https://scholar.google.com/citations?user=TlPI8yIAAAAJ">Zoran Kostic</a> <br/> <a href="https://www.aidl.ee.columbia.edu/">AIDL Lab</a>, <a href="https://www.ee.columbia.edu/">Department of Electrical Engineering</a> <br/> <b>Columbia University</b> <br/><br/> <a href="https://openaccess.thecvf.com/content/CVPR2026W/DriveX/papers/Turkcan_Calibration-Free_View-Agnostic_Monocular_3D_Object_Detection_for_Urban_Scenes_CVPRW_2026_paper.pdf"><img src="https://img.shields.io/badge/Paper-CVF%20Open%20Access-b31b1b.svg" alt="Paper"></a> <a href="https://github.com/mkturkcan/urbanomnidetect"><img src="https://img.shields.io/badge/GitHub-Code-181717.svg?logo=github" alt="GitHub Code"></a> <a href="https://huggingface.co/datasets/mehmetkeremturkcan/UrbanOmniView"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-green" alt="UrbanOmniView Dataset"></a> <a href="https://drivex-workshop.github.io/cvpr2026/"><img src="https://img.shields.io/badge/CVPR%202026-DriveX%20Workshop%20(Oral)-4b44ce.svg" alt="CVPR 2026 DriveX Workshop"></a> <a href="https://www.gnu.org/licenses/agpl-3.0"><img src="https://img.shields.io/badge/License-AGPL%203.0-orange.svg" alt="License"></a> </p> </p> <p align="center"> <img src="https://github.com/mkturkcan/urbanomnidetect/raw/main/assets/urbanomniview.png" width="100%" alt="UrbanOmniView examples across ego-vehicle, infrastructure, and aerial viewpoints"/> </p>

Looking for the latest model? UrbanOmniDetect-2 is a separate follow-up release: one network for COCO-80 detection and calibration-free 3D cuboids. Its checkpoints are hosted in their own repository and are not interchangeable with the ones below.

This repository hosts the pretrained checkpoints from the CVPR 2026 DriveX paper. Training, evaluation, and inference code (single-image BEV, real-time video, and sliced high-resolution tracking) is in the GitHub repository.

Highlights

• Calibration-free 3D detection. A single model predicts 3D bounding box keypoints from a raw RGB image without camera intrinsics, depth estimation, or ground-plane priors.
• View-agnostic. One architecture works across ego-vehicle, infrastructure, and aerial drone viewpoints.
• Strongest on the harder KITTI splits. AP_3D of 30.71 and AP_BEV of 35.19 on the Moderate split at IoU ≥ 0.7, ahead of calibration-dependent baselines on Moderate and Hard.
• Real-time. Under 11 ms per image on an A100 with TensorRT at 640 × 640.
• Robust to calibration error. Calibration-dependent methods lose more than 80% of their accuracy with a 5% focal-length error. UrbanOmniDetect takes no intrinsics as input and is invariant to such errors by construction.

KITTI Benchmark

Monocular 3D detection on KITTI at IoU ≥ 0.7. The baselines use camera calibration. UrbanOmniDetect does not.

Checkpoints

Checkpoints are named urbanomnidetect_<backbone><-head>_<imgsz>.pt and cover YOLOv8, YOLOv9, YOLO11, and YOLO12 across all scales (n, s, m, l, x) and head configurations (default, P1, P2, P6). The P2 feature-pyramid head is the most accurate, and the YOLO11x-P2 family is recommended.

The Files and versions tab lists every checkpoint.

Architecture comparison

Keypoint mAP_50:95 on the UrbanOmniView validation set at 640 × 640. Adding the P2 feature level improves every scale.

Quick Start

Install the dependencies:

pip install ultralytics scipy scikit-learn opencv-python matplotlib

Download the recommended checkpoint:

huggingface-cli download mehmetkeremturkcan/UrbanOmniDetect \
    checkpoints/urbanomnidetect_yolo11x-p2_1920.pt \
    --local-dir .

Or download every checkpoint at once:

huggingface-cli download mehmetkeremturkcan/UrbanOmniDetect --local-dir .

Single-image inference

The model uses the standard Ultralytics prediction API. Point it at an image from any viewpoint and it predicts 3D bounding box keypoints without camera parameters.

from ultralytics import YOLO

model = YOLO("checkpoints/urbanomnidetect_yolo11x-p2_1920.pt")
results = model.predict("your_image.jpg", imgsz=1920, conf=0.1, device="cuda:0")

Each detection has eight ordered keypoints, the 2D projections of its 3D bounding box corners. Indices 0 to 3 are the top corners and indices 4 to 7 are the ground-contact corners.

Bird's-eye view and real-time video

The bird's-eye view, real-time tracking, and sliced high-resolution pipelines (draw_bev.py, bev_realtime.py, sahi_tracker.py) are in the GitHub repository. Ground-contact keypoints are mapped to a top-down plane through an orthogonality-constrained homography, so no camera calibration is needed.

python bev_realtime.py \
    --input drone_manhattan.mp4 \
    --kp-model checkpoints/urbanomnidetect_yolo11x-p2_640.pt \
    --kp-imgsz 640 \
    --aux-model yolo26x.pt \
    --device cuda:0 \
    --export tensorrt \
    --output bev.mp4

Training Data

The models are trained on UrbanOmniView, a 37,446-frame dataset that combines ego-vehicle, infrastructure, and synthetic viewpoints.

The synthetic UE5 portion is released as part of this work.

Citation

If you use UrbanOmniDetect or the UrbanOmniView dataset in your research, please cite:

@inproceedings{turkcan2026urbanomnidetect,
  title     = {Calibration-Free View-Agnostic Monocular {3D} Object Detection for Urban Scenes},
  author    = {Turkcan, Mehmet Kerem and Gumaste, Devika and Kostic, Zoran},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  pages     = {786--795},
  year      = {2026}
}

Acknowledgements

This work began while the first author was a postdoc in the Department of Electrical Engineering (AIDL Lab) at Columbia University. It was supported by the NSF Engineering Research Center for Smart Streetscapes under Award EEC-2133516, NSF Grants CNS-2450567 and CNS-2038984, and by computing resources from the NVIDIA Academic Grant Program and the Empire AI Consortium.

License

Released under the GNU Affero General Public License v3.0.