zipdepth-promptda

ZipDepth-PromptDA v4

A 6.95 M-parameter CNN student for prompted metric depth: (RGB 768×1024, 192×256 metric LiDAR prompt in metres) → metric depth (metres). It is to PromptDA what ZipDepth is to Depth Anything v2: the vendored ZipDepth-base network (unchanged) plus four zero-initialised prompt adapters (Conv3×3 → ReLU → depthwise 3×3 → ReLU → 1×1) injected at the decoder stages f4..f1, and a sigmoid head denormalised by the prompt's depth range so metric scale comes from the LiDAR prompt, not from RGB. Trained purely on PromptDA-large pseudo-labels (2,025 ARKitScenes segments, iPhone RGB + LiDAR) from the released ZipDepth weights.

Files

Numbers (RTX 5090)

Known weakness: thin structures (foliage) wash out; the ARKitScenes labels are the limit, not the architecture (see the project write-up in rerun-io/examples-monorepo, PR stack #186–#192).

Loading

from safetensors.torch import load_file
from monopriors.models.depth_completion.zipdepth_prompt import ZipDepthPrompt  # rerun-io/examples-monorepo
model = ZipDepthPrompt(); model.load_state_dict(load_file("zipdepth_promptda_v4.safetensors"), strict=True); model.eval()
depth_bhw = model(rgb_bhw3_uint8, prompt_bhw_metres)  # via the CompletionConfig / ZipDepthPromptPredictor path

Training recipe (v4): OneCycle, peak lr 1e-5 (config/100), batch 8, 768×1024, bf16, BatchNorm pinned to eval, resize-only augmentation, flat-frame filter; v4 is the end of a ~1M-step chain (v1 70k → v3 500k → v4 +500k). A 2026-09-03 hill-climb found peak lr 1e-4 reaches equal quality in 2.5–3× fewer steps (--preset fast in the training package).

Training curves, chain checkpoints, and notes

• runs/<checkpoint>/ — the complete TensorBoard event files of every run in the chain (zdpda-v1, zdpda-v3, zdpda-v4, zdpda-v5, catalog-ft-v0): loss, learning rate and gradient-loss curves, the io/* throughput traces of the catalog runs (data wait, decode, frames per second), and the periodic input / ground-truth / prediction image grids. The Hub renders the scalars in the Training metrics tab; open the files in TensorBoard for the images. About 12 GB in total.
• chain/<checkpoint>/ — final_model.pth plus the eval JSONs, resolved config, and train/holdout segment lists of the earlier and side runs. The chain is v1 (70k steps) → v3 (500k) → v4 (+500k, the released weights at the repo root); v5 and catalog-ft-v0 are side experiments that did not beat v4.
• notes/ — the working notes of the project: the training-speed hill-climb (notes/hillclimb, incl. the orchestrator, arm table, and morning report), the edge-sharpening research (notes/edge-research), the PR-stack re-cut tooling (notes/recut), and the simplify/review reports (notes/simplify-reports). Text only.