depth-any-camera

---

license: cc-by-nc-sa-2.0

<div align="center"> <h1>Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera</h1>

Yuliang Guo^1*† · Sparsh Garg^2† · S. Mahdi H. Miangoleh³ · Xinyu Huang¹ · Liu Ren¹

¹Bosch Research North America    ²Carnegie Mellon University    ³Simon Fraser University    

 *corresponding author †equal technical contribution

<a href="https://arxiv.org/abs/2501.02464"><img src='https://img.shields.io/badge/arXiv-Depth Any Camera-red' alt='Paper PDF'></a> <a href='https://yuliangguo.github.io/depth-any-camera/'><img src='https://img.shields.io/badge/Project_Page-Depth Any Camera-green' alt='Project Page'></a> <a href='https://github.com/yuliangguo/depth_any_camera'><img src='https://img.shields.io/badge/Code-Depth Any Camera-blue' alt='Code'></a>

</div> <p align="center"> <img src="teaser.png" alt="teaser" style="width: 70%;"> </p>

Depth Any Camera (DAC) is a novel zero-shot metric depth estimation framework that extends a perspective-trained model to handle any type of camera with varying FoVs effectively.

Notably, DAC can be trained exclusively on perspective images, yet it generalizes seamlessly to fisheye and 360 cameras without requiring specialized training data. Key features include:

1. Zero-shot metric depth estimation on fisheye and 360 images, significantly outperforming prior metric depth SoTA Metric3D-v2 and UniDepth.
2. Geometry-based training framework adaptable to any network architecture, extendable to other 3D perception tasks.

Tired of collecting new data for specific cameras? DAC maximizes the utility of every existing 3D data for training, regardless of the specific camera types used in new applications.

Visualization

ScanNet++ fisheye

The zero-shot metric depth estimation results of Depth Any Camera (DAC) are visualized on ScanNet++ fisheye videos and compared to Metric3D-v2. The visualizations of A.Rel error against ground truth highlight the superior performance of DAC.

<p align="center"> <img src="video_scannet++_1.gif" alt="animated" /> </p>

Matterport3D single-view reconstruction

Additionally, we showcase DAC's application on 360-degree images, where a single forward pass of depth estimation enables full 3D scene reconstruction.

<p align="center"> <img src="video_matterport3d_1.gif" alt="animated" /> </p>

Additional visual results and comparison with the prior SoTA can be found at <a href='https://yuliangguo.github.io/depth-any-camera/'><img src='https://img.shields.io/badge/Project_Page-Depth Any Camera-green' alt='Project Page'></a>

Performance

Depth Any Camera performs <b>significantly better</b> than the previous SoTA <b>metric</b> depth estimation models Metric3D-v2 and UniDepth in zero-shot generalization to large FoV camera images given <b>significantly smaller training dataset and model size</b>.

We highlight the best and second best results in bold and italic respectively (better results: AbsRel $\downarrow$ , $\delta_1 \uparrow$).