instant-ngp

Instant Neural Graphics Primitives

<img src="docs/assets_readme/fox.gif" height="342"/> <img src="docs/assets_readme/robot5.gif" height="342"/>

Ever wanted to train a NeRF model of a fox in under 5 seconds? Or fly around a scene captured from photos of a factory robot? Of course you have!

Here you will find an implementation of four neural graphics primitives, being neural radiance fields (NeRF), signed distance functions (SDFs), neural images, and neural volumes. In each case, we train and render a MLP with multiresolution hash input encoding using the tiny-cuda-nn framework.

Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
Thomas Müller, Alex Evans, Christoph Schied, Alexander Keller
ACM Transactions on Graphics (SIGGRAPH), July 2022
Project page / Paper / Video / Presentation / Real-Time Live / BibTeX

To get started with NVIDIA Instant NeRF, check out the blog post and SIGGRAPH tutorial.

For business inquiries, please submit the NVIDIA research licensing form.

Windows binary release

If you have Windows and if you do not need Python bindings, you can download one of the following binary releases and then jump directly to the usage instructions. These releases are automatically regenerated whenever the code gets updated, so you can be sure that they have the latest features.

• RTX 3000 & 4000 series, RTX A4000–A6000, and other Ampere & Ada cards
• RTX 2000 series, Titan RTX, Quadro RTX 4000–8000, and other Turing cards
• GTX 1000 series, Titan Xp, Quadro P1000–P6000, and other Pascal cards

If you use Linux, or want the Python bindings, or if your GPU is not listed above (e.g. Hopper, Volta, or Maxwell generations), use the following step-by-step instructions to compile instant-ngp yourself.

Requirements

• An NVIDIA GPU; tensor cores increase performance when available. All shown results come from an RTX 3090.
• A C++14 capable compiler. The following choices are recommended and have been tested:
  • Windows: Visual Studio 2019 or 2022
  • Linux: GCC/G++ 8 or higher
• A recent version of CUDA. The following choices are recommended and have been tested:
  • Windows: CUDA 11.5 or higher
  • Linux: CUDA 10.2 or higher
• CMake v3.21 or higher.
• (optional) Python 3.7 or higher for interactive bindings. Also, run pip install -r requirements.txt.
• (optional) OptiX 7.3 or higher for faster mesh SDF training.
• (optional) Vulkan SDK for DLSS support.

If you are using Debian based Linux distribution, install the following packages

sudo apt-get install build-essential git python3-dev python3-pip libopenexr-dev libxi-dev \
                     libglfw3-dev libglew-dev libomp-dev libxinerama-dev libxcursor-dev

Alternatively, if you are using Arch or Arch derivatives, install the following packages

sudo pacman -S base-devel cmake openexr libxi glfw openmp libxinerama libxcursor

We also recommend installing CUDA and OptiX in /usr/local/ and adding the CUDA installation to your PATH.

For example, if you have CUDA 11.4, add the following to your ~/.bashrc

export PATH="/usr/local/cuda-11.4/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-11.4/lib64:$LD_LIBRARY_PATH"

For Arch and derivatives,

sudo pacman -S cuda

Compilation (Windows & Linux)

Begin by cloning this repository and all its submodules using the following command:

$ git clone --recursive https://github.com/nvlabs/instant-ngp
$ cd instant-ngp

Then, use CMake to build the project: (on Windows, this must be in a developer command prompt)

instant-ngp$ cmake . -B build
instant-ngp$ cmake --build build --config RelWithDebInfo -j

If compilation fails inexplicably or takes longer than an hour, you might be running out of memory. Try running the above command without -j in that case. If this does not help, please consult this list of possible fixes before opening an issue.

If the build succeeds, you can now run the code via the build/instant-ngp executable or the scripts/run.py script described below.

If automatic GPU architecture detection fails, (as can happen if you have multiple GPUs installed), set the TCNN_CUDA_ARCHITECTURES environment variable for the GPU you would like to use. The following table lists the values for common GPUs. If your GPU is not listed, consult this exhaustive list.

Interactive training and rendering

<img src="docs/assets_readme/testbed.png" width="100%"/>

This codebase comes with an interactive GUI that includes many features beyond our academic publication:

• Additional training features, such as extrinsics and intrinsics optimization.
• Marching cubes for NeRF->Mesh and SDF->Mesh conversion.
• A spline-based camera path editor to create videos.
• Debug visualizations of the activations of every neuron input and output.
• And many more task-specific settings.
• See also our one minute demonstration video of the tool.

Let's start using instant-ngp; more information about the GUI and other scripts follow these test scenes.

NeRF fox

One test scene is provided in this repository, using a small number of frames from a casually captured phone video:

instant-ngp$ ./build/instant-ngp --scene data/nerf/fox

On Windows you need to reverse the slashes here (and below), i.e.:

instant-ngp> .\build\instant-ngp --scene data\nerf\fox

<img src="docs/assets_readme/fox.png"/>

Alternatively, download any NeRF-compatible scene (e.g. from the NeRF authors' drive, the SILVR dataset, or the DroneDeploy dataset). Now you can run:

instant-ngp$ ./build/instant-ngp --scene data/nerf_synthetic/lego/transforms_train.json

To prepare your own dataset for use with our NeRF implementation, click here. See also this video for a guided walkthrough.

SDF armadillo

instant-ngp$ ./build/instant-ngp --scene data/sdf/armadillo.obj

<img src="docs/assets_readme/armadillo.png"/>

Image of Einstein

instant-ngp$ ./build/instant-ngp --scene data/image/albert.exr

<img src="docs/assets_readme/albert.png"/>

To reproduce the gigapixel results, download, for example, the Tokyo image and convert it to .bin using the scripts/convert_image.py script. This custom format improves compatibility and loading speed when resolution is high. Now you can run:

instant-ngp$ ./build/instant-ngp --scene data/image/tokyo.bin

Volume renderer

Download the nanovdb volume for the Disney cloud, which is derived from here (CC BY-SA 3.0).

instant-ngp$ ./build/instant-ngp --mode volume --scene data/volume/wdas_cloud_quarter.nvdb

<img src="docs/assets_readme/cloud.png"/>

GUI controls

Here are the main keyboard controls for the instant-ngp application.

There are many controls in the instant-ngp GUI. First, note that this GUI can be moved and resized, as can the "Camera path" GUI (which first must be expanded to be used).

Some popular user controls in instant-ngp are:

• Snapshot: use Save to save the NeRF solution generated, Load to reload. Necessary if you want to make an animation.
• Rendering -> DLSS: toggling this on and setting "DLSS sharpening" below it to 1.0 can often improve rendering quality.
• Rendering -> Crop size: trim back the surrounding environment to focus on the model. "Crop aabb" lets you move the center of the volume of interest and fine tune. See more about this feature in our NeRF training & dataset tips.

The "Camera path" GUI lets you set frames along a path. "Add from cam" is the main button you'll want to push, then saving out the camera keyframes using "Save" to create a base_cam.json file. There is a bit more information about the GUI in this post and in this (bit dated) video.

Python bindings

To conduct controlled experiments in an automated fashion, all features from the interactive GUI (and more!) have Python bindings that can be easily instrumented. For an example of how the ./build/instant-ngp application can be implemented and extended from within Python, see ./scripts/run.py, which supports a superset of the command line arguments that ./build/instant-ngp does.

Here is a typical command line using scripts/run.py to generate a 5-second flythrough of the fox dataset to the (default) file video.mp4, after using the GUI to save a (default) NeRF snapshot base.msgpack and a set of camera key frames: (see this video for a guided walkthrough)

instant-ngp$ python scripts/run.py --mode nerf --scene data/nerf/fox --load_snapshot data/nerf/fox/base.msgpack --video_camera_path data/nerf/fox/base_cam.json --video_n_seconds 5 --video_fps 60 --width 1920 --height 1080

If you'd rather build new models from the hash encoding and fast neural networks, consider the tiny-cuda-nn's PyTorch extension.

Happy hacking!

Frequently asked questions (FAQ)

Q: How can I run instant-ngp in headless mode?

A: Use ./build/instant-ngp --no-gui or python scripts/run.py. You can also compile without GUI via cmake -DNGP_BUILD_WITH_GUI=off ...

Q: Does this codebase run on Google Colab?

A: Yes. See this example inspired on the notebook created by user @myagues. Caveat: this codebase requires large amounts of GPU RAM and might not fit on your assigned GPU. It will also run slower on older GPUs.

Q: Is there a Docker container?

A: Yes. We bundle a Visual Studio Code development container, the .devcontainer/Dockerfile of which you can also use stand-alone.

If you want to run the container without using VSCode:

docker-compose -f .devcontainer/docker-compose.yml build instant-ngp
xhost local:root
docker-compose -f .devcontainer/docker-compose.yml run instant-ngp /bin/bash

Then run the build commands above as normal.

Q: How can I edit and train the underlying hash encoding or neural network on a new task?

A: Use tiny-cuda-nn's PyTorch extension.

Q: How can I save the trained model and load it again later?

A: Two options:

1. Use the GUI's "Snapshot" section.
2. Use the Python bindings load_snapshot / save_snapshot (see scripts/run.py for example usage).

Q: Can this codebase use multiple GPUs at the same time?

A: No. To select a specific GPU to run on, use the CUDA_VISIBLE_DEVICES environment variable. To optimize the compilation for that specific GPU use the TCNN_CUDA_ARCHITECTURES environment variable.

Q: What is the coordinate system convention?

A: See this helpful diagram by user @jc211.

Q: The NeRF reconstruction of my custom dataset looks bad; what can I do?

A: There could be multiple issues:

• COLMAP might have been unable to reconstruct camera poses.
• There might have been movement or blur during capture. Don't treat capture as an artistic task; treat it as photogrammetry. You want *as little blur as possible* in your dataset (motion, defocus, or otherwise) and all objects must be *static* during the entire capture. Bonus points if you are using a wide-angle lens (iPhone wide angle works well), because it covers more space than narrow lenses.
• The dataset parameters (in particular aabb_scale) might have been tuned suboptimally. We recommend starting with aabb_scale=16 and then increasing or decreasing it by factors of two until you get optimal quality.
• Carefully read our NeRF training & dataset tips.

Q: Why are background colors randomized during NeRF training?

A: Transparency in the training data indicates a desire for transparency in the learned model. Using a solid background color, the model can minimize its loss by simply predicting that background color, rather than transparency (zero density). By randomizing the background colors, the model is forced to learn zero density to let the randomized colors "shine through".

Q: How to mask away NeRF training pixels (e.g. for dynamic object removal)?

A: For any training image xyz.* with dynamic objects, you can provide a dynamic_mask_xyz.png in the same folder. This file must be in PNG format, where non-zero pixel values indicate masked-away regions.

Troubleshooting compile errors

Before investigating further, make sure all submodules are up-to-date and try compiling again.

instant-ngp$ git submodule sync --recursive
instant-ngp$ git submodule update --init --recursive

If instant-ngp still fails to compile, update CUDA as well as your compiler to the latest versions you can install on your system. It is crucial that you update both, as newer CUDA versions are not always compatible with earlier compilers and vice versa. If your problem persists, consult the following table of known issues.

*After each step, delete the build folder and let CMake regenerate it before trying again.*

If you cannot find your problem in the table, try searching the discussions board and the issues area for help. If you are still stuck, please open an issue and ask for help.

Thanks

Many thanks to Jonathan Tremblay and Andrew Tao for testing early versions of this codebase and to Arman Toorians and Saurabh Jain for the factory robot dataset. We also thank Andrew Webb for noticing that one of the prime numbers in the spatial hash was not actually prime; this has been fixed since.

This project makes use of a number of awesome open source libraries, including:

• tiny-cuda-nn for fast CUDA networks and input encodings
• tinyexr for EXR format support
• tinyobjloader for OBJ format support
• stb_image for PNG and JPEG support
• Dear ImGui an excellent immediate mode GUI library
• Eigen a C++ template library for linear algebra
• pybind11 for seamless C++ / Python interop
• and others! See the dependencies folder.

Many thanks to the authors of these brilliant projects!

License and Citation

@article{mueller2022instant,
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2022},
    volume = {41},
    number = {4},
    month = jul,
    year = {2022},
    pages = {102:1--102:15},
    articleno = {102},
    numpages = {15},
    url = {https://doi.org/10.1145/3528223.3530127},
    doi = {10.1145/3528223.3530127},
    publisher = {ACM},
    address = {New York, NY, USA},
}

Copyright © 2022, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License-NC. Click here to view a copy of this license.