wild-gaussians

<p align="center"> <h1 align="center">WildGaussians: 3D Gaussian Splatting in the Wild</h1> <p align="center"> <a href="https://jkulhanek.com/">Jonas Kulhanek</a> · <a href="https://pengsongyou.github.io/">Songyou Peng</a> · <a href="https://cmp.felk.cvut.cz/~kukelova/">Zuzana Kukelova</a> · <a href="https://people.inf.ethz.ch/marc.pollefeys/">Marc Pollefeys</a> · <a href="https://tsattler.github.io/">Torsten Sattler</a> </p> <h3 align="center"><a href="https://arxiv.org/pdf/2407.08447">📄 Paper</a> | <a href="https://wild-gaussians.github.io/">🌐 Project Page</a> | <a href="https://github.com/jkulhanek/wild-gaussians">🖥 Code</a></a></h3> </p> <p align="center"> <table> <tr> <td width="10%"></td> <td width="40.8%"> <img width="100%" alt="WildGaussians model appearance" src="https://github.com/jkulhanek/wild-gaussians/raw/main/.assets/cover-trevi.webp" /> </td> <td width="34.4%"> <img width="100%" alt="WildGaussians remove occluders" src="https://github.com/jkulhanek/wild-gaussians/raw/main/.assets/cover-onthego.webp" /> </td> <td width="10%"></td> </tr> </table> </p> <p align="justify"> We introduce WildGaussians, a novel approach to handle occlusions and appearance changes with 3DGS. By leveraging robust DINO features and integrating an appearance modeling module within 3DGS, our method achieves state-of-the-art results. We demonstrate that WildGaussians matches the real-time rendering speed of 3DGS while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all within a simple architectural framework. </p> <br>

Model Details

• Developed by: CTU in Prague, ETH Zurich
• License: MIT
• Repository: https://github.com/jkulhanek/wild-gaussians
• Web: https://wild-gaussians.github.io
• Paper: https://arxiv.org/pdf/2407.08447

Installation

We offer two ways to install the method: local installation (through a conda environment) and installation from NerfBaselines, where multiple backends (conda,docker,apptainer) can be selected.

Install from NerfBaselines (recommended)

The easiest way to install WildGaussians is by using NerfBaselines. First (if not already done), install NerfBaselines by running:

conda create -n nb python=3.11
conda activate nb
pip install nerfbaselines>=1.0.1

WildGaussians is then installed by running:

nerfbaselines install-method --spec https://raw.githubusercontent.com/jkulhanek/wild-gaussians/main/wildgaussians/wildgaussians_spec.py

Local installation

Clone the repository and create a python == 3.11 Anaconda environment with CUDA toolkit 11.8 installed using

git clone [email protected]:jkulhanek/wild-gaussians.git
cd wild-gaussians

conda create -y -n wg python=3.11
conda activate wg
conda install -y --override-channels -c nvidia/label/cuda-11.8.0 cuda-toolkit
conda env config vars set NERFBASELINES_BACKEND=python
pip install --upgrade pip
pip install -r requirements.txt
pip install nerfbaselines>=1.0.1
pip install -e ./submodules/diff-gaussian-rasterization ./submodules/simple-knn
pip install -e .

Checkpoints, predictions, and data

The checkpoints and predictions are available online at https://huggingface.co/jkulhanek/wild-gaussians/tree/main:

<ul> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip?download=true">Photo Tourism/Trevi Fountain</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip?download=true">Photo Tourism/Sacre Coeur</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip?download=true">Photo Tourism/Brandenburg Gate</a></li> <br> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/fountain.zip?download=true">NeRF On-the-go/Fountain</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/mountain.zip?download=true">NeRF On-the-go/Mountain</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/corner.zip?download=true">NeRF On-the-go/Corner</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio.zip?download=true">NeRF On-the-go/Patio</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/spot.zip?download=true">NeRF On-the-go/Spot</a></li> <li><a href="https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio-high.zip?download=true">NeRF On-the-go/Patio High</a></li> </ul>

In order to train/evaluate on the NeRF On-the-go dataset, please download the undistorted version from the following link: https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main

Interactive viewer

To start the viewer and explore the trained models, run one of the following:

# Photo Tourism
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/trevi-fountain.zip/checkpoint --data external://phototourism/trevi-fountain
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/sacre-coeur.zip/checkpoint --data external://phototourism/sacre-coeur
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/phototourism/brandenburg-gate.zip/checkpoint --data external://phototourism/brandenburg-gate

# NeRF on-the-go
# NOTE: Optionally attach `--data {path to data}` to load the dataset (recommended)
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/fountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/mountain.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/corner.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/spot.zip/checkpoint
nerfbaselines viewer --checkpoint https://huggingface.co/jkulhanek/wild-gaussians/resolve/main/nerfonthego/patio-high.zip/checkpoint

Training

To start the training on the Photo Tourism dataset, run one of following commands:

# Photo Tourism
nerfbaselines train --method wildgaussians --data external://phototourism/trevi-fountain
nerfbaselines train --method wildgaussians --data external://phototourism/sacre-coeur
nerfbaselines train --method wildgaussians --data external://phototourism/brandenburg-gate

# NeRF on-the-go
nerfbaselines train --method wildgaussians --data {path to data}

The training will also generate output artifacts containing the test set predictions, checkpoint, and tensorboard logs.

Rendering videos

To render a video on a trajectory (e.g., generated from the interactive viewer), run:

nerfbaselines render --checkpoint {checkpoint} --trajectory {trajectory file}

Concurrent works

There are several concurrent works that also aim to extend 3DGS to handle in-the-wild data:

<ul> <li>Wild-GS: Real-Time Novel View Synthesis from Unconstrained Photo Collections</li> <li>Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections</li> <li>SpotlessSplats: Ignoring Distractors in 3D Gaussian Splatting</li> <li>SWAG: Splatting in the Wild images with Appearance-conditioned Gaussians</li> <li>WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections</li> </ul> </p> </section>

Acknowledgements

We would like to thank Weining Ren for his help with the NeRF On-the-go dataset and code and Tobias Fischer and Xi Wang for fruitful discussions. This work was supported by the Czech Science Foundation (GAČR) EXPRO (grant no. 23-07973X) and by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254). The renderer is built on 3DGS and Mip-Splatting. Please follow the license of 3DGS and Mip-Splatting. We thank all the authors for their great work and released code.

Citation

If you find our code or paper useful, please cite:

@article{kulhanek2024wildgaussians,
  title={WildGaussians: 3D Gaussian Splatting in the Wild},
  author={Kulhanek, Jonas and Peng, Songyou and Kukelova, Zuzana and Pollefeys, Marc and Sattler, Torsten},
  journal={arXiv},
  year={2024}
}