zoedepth-nyu-kitti-onnx

ZoeDepth — Metric Monocular Depth (ONNX)

ONNX export of Intel/zoedepth-nyu-kitti — Intel ISL's metric-depth follow-up to DPT-Large. Same DPT-Large backbone with calibrated metric-bin heads grafted on (one trained on NYU indoor depths, one on KITTI outdoor depths, combined via a domain-routing classifier). Outputs depth in real-world meters, not just relative ordering.

Re-exported from upstream PyTorch weights — Intel publishes only safetensors at the source repo. Provenance trail: Bhat et al. → Intel/zoedepth-nyu-kitti → transformers.ZoeDepthForDepthEstimation + thin wrapper → torch.onnx.export → these files. fp16 sibling produced from the fp32 trace via onnxconverter-common.

Toolchain: torch 2.4.x (CUDA 12.4), torchvision 0.19 (matched ABI), transformers 4.45.2, optimum[onnxruntime] 1.24.0, onnxconverter-common>=1.14, opset 17, do_constant_folding=True. Full conversion script: scripts/export-zoedepth.ps1 in the Heliosoph repo (run once for fp32, again with -Fp16 for the half-precision sibling).

Credit: Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, Matthias Müller (Intel Intelligent Systems Lab). Paper: "ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth", 2023.

What this repo contains

The fp32 model.onnx is self-contained — it came in at 1.37 GB, just under the 2 GB protobuf limit, so no external-data .onnx.data sidecar. Same for the fp16 variant at 693 MB.

What "metric depth" means (vs the other depth models on Heliosoph)

Pick this repo specifically when you need meters — most monocular depth models give you a number per pixel that's only meaningful relative to other pixels in the same image; this one gives you a number that's calibrated against real-world distance.

Input / output

How to use

import onnxruntime as ort
import numpy as np
from PIL import Image
from transformers import ZoeDepthImageProcessor

# Use the included preprocessor — it handles the 32-alignment + normalize.
proc = ZoeDepthImageProcessor.from_pretrained(".")
sess = ort.InferenceSession("model.onnx")          # or "model_fp16.onnx"

img = Image.open("photo.jpg").convert("RGB")
inputs = proc(images=img, return_tensors="np")     # NCHW float32, 32-aligned

depth_meters = sess.run(
    None,
    {"pixel_values": inputs["pixel_values"]},
)[0][0]                                            # [H, W], meters

# depth_meters[y, x] = distance from camera to that surface point, in meters

For the fp16 model, the input also needs to be float16 — cast inputs["pixel_values"] to np.float16 before feeding it in.

Why two variants

• fp32 is the safe default — identical numerics to the upstream PyTorch reference, no surprises.
• fp16 halves disk footprint and model-load memory. On GPU / NPU with native fp16 you also get a modest speedup; on CPU runtimes that upcast fp16→fp32 internally the speed is identical to fp32 but you save the memory. Depth output is essentially identical (the fp16 quantization noise is below the model's own per-pixel error).

If you're not sure: pick fp32 for accuracy-sensitive scientific work, fp16 for shipping / deployment / edge.

License

MIT — same as upstream Intel/zoedepth-nyu-kitti. LICENSE file included. The ONNX-export step (and the fp16 numerical conversion) doesn't change licensing — same model, different serialization.