gim

<div style="display: flex; justify-content: center; align-items: center;"> <div style="width: 100%; text-align: center; font-size: 26px;"> <strong>GIM: Learning Generalizable Image Matcher From Internet Videos</strong> </div> </div> <table cellspacing="0" cellpadding="0" style="margin: 0;"> <tr> <td><a href="https://iclr.cc/Conferences/2024" target="_blank"><img src="https://img.shields.io/badge/%F0%9F%8C%9F_ICLR'2024_Spotlight-37414c" alt="ICLR 2024 Spotlight" style="margin: 0;"></a></td> <td><a href="https://xuelunshen.com/gim" target="_blank"><img src="https://img.shields.io/badge/Project_Page-3A464E?logo=gumtree" alt="Project Page" style="margin: 0;"></a></td> <td><a href="https://github.com/xuelunshen/gim" target="_blank"><img src="https://img.shields.io/badge/Source_Code-black?logo=Github" alt="Github Source Code" style="margin: 0;"></a></td> <td><a href="https://arxiv.org/abs/2402.11095" target="_blank"><img src="https://img.shields.io/badge/arXiv-2402.11095-b31b1b?logo=arxiv" alt="arxiv" style="margin: 0;"></a></td> <td><a href="https://www.youtube.com/watch?v=FU_MJLD8LeY" target="_blank"><img src="https://img.shields.io/badge/Overview_Video-E33122?logo=Youtube" alt="Overview Video" style="margin: 0;"></a></td> </tr> </table> <div align="center"><a href="https://www.youtube.com/watch?v=FU_MJLD8LeY" target="_blank"><img src="video.png" alt="Overview Video Page" style="margin: 0; width: 75%;"></a></div> <div style="display: flex; justify-content: center; padding: 10px; align-items: left;"> <div style="width: 100%; text-align: left; background-color: #FFFFFF;"> We propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. </div> </div> <div style="display: flex; justify-content: center; padding: 10px; align-items: left;"> <div style="width: 100%; text-align: left; background-color: #FFFFFF;"> Image matching is a fundamental computer vision problem. While learning-based methods achieve state-of-the-art performance on existing benchmarks, they generalize poorly to in-the-wild images. Such methods typically need to train separate models for different scene types (e.g., indoor vs. outdoor) and are impractical when the scene type is unknown in advance. One of the underlying problems is the limited scalability of existing data construction pipelines, which limits the diversity of standard image matching datasets. To address this problem, we propose GIM, a self-training framework for learning a single generalizable model based on any image matching architecture using internet videos, an abundant and diverse data source. Given an architecture, GIM first trains it on standard domain-specific datasets and then combines it with complementary matching methods to create dense labels on nearby frames of novel videos. These labels are filtered by robust fitting, and then enhanced by propagating them to distant frames. The final model is trained on propagated data with strong augmentations. Not relying on complex 3D reconstruction makes GIM much more efficient and less likely to fail than standard SfM-and-MVS based frameworks. </div> </div> <div style="display: flex; justify-content: center; padding: 10px; align-items: left;"> <div style="width: 100%; text-align: left; background-color: #FFFFFF;"> We also propose ZEB, the first zero-shot evaluation benchmark for image matching. By mixing data from diverse domains, ZEB can thoroughly assess the cross-domain generalization performance of different methods. </div> </div> <div style="display: flex; justify-content: center; padding: 10px; align-items: left;"> <div style="width: 100%; text-align: left; background-color: #FFFFFF;"> Experiments demonstrate the effectiveness and generality of GIM. Applying GIM consistently improves the zero-shot performance of 3 state-of-the-art image matching architectures as the number of downloaded videos increases; with 50 hours of YouTube videos, the relative zero-shot performance improves by 8.4% − 18.1%. GIM also enables generalization to extreme cross-domain data such as Bird Eye View (BEV) images of projected 3D point clouds. More importantly, our single zero-shot model consistently outperforms domain-specific baselines when evaluated on downstream tasks inherent to their respective domains. </div> </div> </br>

Usage

We recommend checking out Huggingface online demo https://huggingface.co/spaces/xuelunshen/gim-online and Github Repository: https://github.com/xuelunshen/gim, which provides code for finding matches from any image pairs.

Citation

BibTeX:

@inproceedings{
xuelun2024gim,
title={GIM: Learning Generalizable Image Matcher From Internet Videos},
author={Xuelun Shen and Zhipeng Cai and Wei Yin and Matthias Müller and Zijun Li and Kaixuan Wang and Xiaozhi Chen and Cheng Wang},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024}
}