loc3r-vlm-llava

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

<p align="center"> <a href="https://kevinqu7.github.io">Kevin Qu</a>^1,2*, <a href="https://scholar.google.com/citations?user=BajLgxUAAAAJ&hl=en">Haozhe Qi</a>³, <a href="https://dusmanu.com/">Mihai Dusmanu</a>¹, <a href="https://radmahdi.github.io/Home.html">Mahdi Rad</a>¹, <a href="https://rui2016.github.io/">Rui Wang</a>¹, <a href="https://people.inf.ethz.ch/pomarc/">Marc Pollefeys</a>^1,2 </p> <p align="center"> ¹Microsoft Spatial AI Lab &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ²ETH Zurich &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ³EPFL </p> <p align="center"> <small>*work done during an internship at Microsoft</small> </p> <p align="center"> <a href="https://kevinqu7.github.io/loc3r-vlm"> <img src="https://img.shields.io/badge/Project_Page-Loc3R--VLM-blue" alt="Project Page"> </a> <a href="https://arxiv.org/abs/2603.18002"> <img src="https://img.shields.io/badge/arXiv-2603.18002-blue?logo=arxiv&color=%23B31B1B" alt="arXiv"> </a> <a href="https://github.com/kevinqu7/Loc3R-VLM"> <img src="https://img.shields.io/badge/GitHub-Repository-black?logo=github" alt="GitHub"> </a> </p>

We introduce Loc3R-VLM, a novel framework that equips 2D VLMs with advanced 3D spatial understanding capabilities from video. Inspired by human cognition, it builds an internal cognitive map of the global environment while explicitly modeling an agent's position and orientation. By jointly capturing global layout and egocentric state, the model excels at two core tasks: language-driven localization and viewpoint-aware 3D reasoning.

This repository hosts the released model checkpoint, which uses LLaVA-Video-7B-Qwen2 as the base model.

More details can be found in our GitHub repository.

Citation

@article{qu2026loc3rvlm, 
    title={Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models},
    author={Kevin Qu and Haozhe Qi and Mihai Dusmanu and Mahdi Rad and Rui Wang and Marc Pollefeys}, 
    journal={arXiv preprint arXiv:2603.18002}, 
    year={2026} 
}