Downloads · 30 days
10
9% of all-time downloads
Lotus-Depth-D-ONNX
Jens-Duttke/Lotus-Depth-D-ONNX
Lotus-Depth-D-ONNX is a depth estimation model from Jens-Duttke. Use it for the depth estimation task on the model card, and read the license before you ship it in a product. It is set up for onnxruntime. The card lists the license as apache-2.0.
Single-graph ONNX build of Lotus-D depth, a diffusion-based dense prediction model, packaged for stereo and disparity workflows.
All-time downloads
112
Public
Repo size
3.9 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.onnx1.9 GB · 98%
At a glance
- Task
- Depth Estimation
- Library
- onnxruntime
- License
- apache-2.0
- Model type
- depth-estimation
- Access
- Public
- Created
- Aug 8, 2026
- Updated
- Aug 21, 2026
- SHA
- a9535ca6
Base models
Model card
From the Hugging Face model README
Lotus-D Depth - ONNX
Single-graph ONNX build of Lotus-D depth, a diffusion-based dense prediction model, packaged for stereo and disparity workflows.
Lotus is derived from Stable Diffusion 2.1 but is not a diffusion pipeline at inference: it collapses the formulation to a single training time-step, fixes t = T, and the "-d" (discriminative) variant drops the Gaussian noise input entirely. One deterministic forward pass per image, no scheduler, no ensembling. This build folds the whole chain - VAE encoder, UNet, VAE decoder - into one ONNX file with one input and one output.
The output is a ready-to-use disparity map (near = bright, far = dark) - no postprocessing, no inversion.
[!TIP] See it in action: Oku3D Media Player converts any 2D video or photo into immersive 3D for autostereoscopic and lenticular displays - "Watch everything in 3D."
Key Features
- Deterministic, single-pass: no scheduler loop, no ensembling, no per-frame seed. The same image always yields the same depth map, which matters for video, where a resampled latent would show up as flicker.
- Everything constant is baked in: the empty-prompt CLIP embedding, the depth task embedding, and the timestep are graph initializers. There is no text encoder to ship and no second input to feed.
- Disparity-native: this checkpoint is trained in disparity space, so near = high / far = low comes straight out of the model. No transfer function, no risk of an inverted 3D result.
- Any resolution that is a multiple of 64:
heightandwidthare symbolic, and one graph serves 256 through 1024 - verified on both the CPU and DirectML providers. - FP32 in, FP32 out, FP16 weights: standard boundary types, half-precision storage. The ImageNet to
[-1, 1]conversion the Stable Diffusion VAE needs happens inside the graph. - Opset 21.
Technical Specifications
| Property | Value |
|---|---|
| Input shape | (1, 3, height, width) NCHW, both axes a multiple of 64 |
| Input dtype | float32 |
| Input range | ImageNet-normalized RGB (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]) |
| Output shape | (1, 1, height, width) |
| Output dtype | float32 |
| Output range | [0, 5] disparity (higher = closer) |
| Batch size | fixed at 1 |
| Parameters | 951M (UNet 868M, VAE encoder 34M, VAE decoder 50M) |
| Opset | 21 |
Sizes must be a multiple of 64
The VAE encoder floors its input to a latent grid of floor(edge / 8), and the UNet halves that three more times, so it needs a latent divisible by 8. Two failure modes follow from that, and neither raises a helpful error:
edge % 64between 1 and 7 - the latent still lands on a multiple of 8, so the graph runs and silently returns the input rounded down to the next multiple of 64.519in,512x512out.- Anything else - the latent is not divisible by 8 and the run fails inside a
Concatin the UNet skip connections.
Being a multiple of 8 decides nothing on its own: 520 is one and fails, 519 is not one and runs. Measured: 448, 455, 512, 518, 519, 576 and 583 all run; 456, 500, 520, 568 and 584 all fail.
Sizes verified working at their own size: 256, 320, 384, 512, 640, 768, 1024.
Files
| File | Weights | Size | Nodes |
|---|---|---|---|
lotus-depth-d_fp16_opset21_optimized.onnx | float16 | 1815 MB | 3422 |
One self-contained file - no external-data companion.
Attention is fused. ONNX Runtime's graph optimizer merges the 16 self-attention blocks into single Attention nodes, which takes the graph from 4710 to 3422 nodes. That matters most where attention dominates: its cost grows with the square of the sequence length, so with the fourth power of the edge. The fused kernel also never materialises the score matrices, which is why memory use plateaus below 5 GB from 768 upward instead of climbing to 13 GB at 1024 - the difference between the top resolution needing a 24 GB card and a mid-range one.
The fusion had been ruled out on the grounds that com.microsoft.MultiHeadAttention returns wrong values on DirectML. Two findings changed that. The unet preset does not emit that op on this graph: it emits Attention, which has a working DirectML kernel. And the reported defect no longer reproduces on a current runtime. Output deviation against the unfused build stays below what DirectML already deviates from the CPU provider - see Performance. That, like every number here, was measured on an AMD adapter; DirectML kernels come from the driver, so another vendor's deviation is its own question.
Why there is no 4-bit build. One was made and measured, and it does not earn its place on this architecture. Only 29% of this graph's weight bytes are in MatMul/Gemm; 68% are in convolutions, which MatMulNBits cannot touch. So the file only drops to 1427 MB, while DirectML throughput falls from 36.8 to 2.9 fps at 256x256 - both measured against the unfused build the experiment was run on - and the mean deviation rises to 0.95-2.35% of the output range. The kernel itself is fine in isolation (a quantized 1280x1280 MatMul runs 22% slower than its FP16 equivalent on the same GPU), so the loss comes from how the quantized nodes schedule inside the full graph. On ViT-based depth models the same recipe pays off; here it does not.
Requirements
- ONNX Runtime 1.18 or newer - the opset floor, since 1.17 tops out at opset 20 - with the
com.microsoftcontrib ops, which every standard build ships. The 16Attentionnodes are the only part of this graph outside the base opset; a runtime compiled with contrib ops disabled will not load it. Verified on 1.23 (DirectML and CPU providers); the consuming application runs the same file on 1.24 DirectML. It also loads on 1.18.1, but that runtime's DirectML provider is 24x slower on this graph, so the figures below need a 1.23-era runtime. - Roughly 2 GB of GPU memory at 256x256, rising to 5 GB at 1024x1024 - see Performance.
DirectML: leave the dimensions symbolic
DirectML's graph fusion faults during session creation whenever this graph's input shape is concrete:
RUNTIME_EXCEPTION : Exception during initialization:
...\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp ... 80070057
That includes pinning height/width through free-dimension overrides, and it applies to statically exported versions of this graph at any resolution too. Just feed the tensor at the size you want; the graph is fully dynamic. If your host insists on pinning, set the session config entry ep.dml.disable_graph_fusion to 1 - it works, at a cost of roughly a quarter of the throughput.
Quick Start
import cv2
import numpy as np
import onnxruntime as ort
MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
SIZE = 512 # any multiple of 64
session = ort.InferenceSession(
"lotus-depth-d_fp16_opset21_optimized.onnx",
providers=["DmlExecutionProvider", "CPUExecutionProvider"],
)
bgr = cv2.imread("image.jpg")
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
rgb = cv2.resize(rgb, (SIZE, SIZE), interpolation=cv2.INTER_AREA)
x = (rgb.astype(np.float32) / 255.0 - MEAN) / STD
x = np.transpose(x, (2, 0, 1))[np.newaxis].astype(np.float32)
disparity = session.run(None, {session.get_inputs()[0].name: x})[0] # (1, 1, S, S)
# near = high; normalize for display
d = disparity.squeeze()
d = (d - d.min()) / max(d.max() - d.min(), 1e-6)
cv2.imwrite("depth.png", (d * 255).astype(np.uint8))
Performance
Benchmarked on an AMD Radeon RX 7900 XTX with ONNX Runtime 1.23 and DirectML, batch size 1, dimensions left symbolic. VRAM is dedicated GPU memory measured on top of the idle baseline.
| Resolution | Throughput | Latency | VRAM |
|---|---|---|---|
| 256x256 | 40.0 fps | 25 ms | 2.2 GB |
| 384x384 | 24.3 fps | 41 ms | 2.7 GB |
| 512x512 | 15.2 fps | 66 ms | 2.7 GB |
| 768x768 | 6.6 fps | 152 ms | 4.8 GB |
| 1024x1024 | 3.4 fps | 298 ms | 4.9 GB |
The latency curve is steep by construction: a UNet at 1024x1024 carries 16x the latent tokens of one at 256x256, and attention is quadratic on top of that. 256 and 384 are the video-capable tiers on this class of GPU; 512 and above are for stills unless the GPU is considerably faster.
Memory does not follow that curve. It plateaus below 5 GB from 768 upward, because the fused attention kernel never materialises the score matrices whose size grows with the square of the sequence length. Without the fusion the same weights need 7.0 GB at 768 and 13.3 GB at 1024, and run 1.4x and 1.8x slower there - so the fusion decides whether the top tier needs a 24 GB card or a mid-range one. The gain is small where attention is not the bottleneck: 6% at 256.
The fused build's output deviates from the unfused one by less than the DirectML provider already deviates from the CPU provider - 0.50% of pixels off by more than one 8-bit step, against a 0.82% provider baseline.
Accuracy against a full-precision PyTorch reference of the same chain, measured on the unfused build of these weights, on the four reference images at 512x512: mean deviation 0.004-0.043% of the output range, rank correlation 0.99997 or better, on both the DirectML and CPU providers. The fusion re-packs the attention projections without changing a value, and its own deviation is the paragraph above.
Comparison: Quality
Side-by-side plasma renders against the strongest variant of each earlier depth-model generation. Plasma convention is uniform (yellow = near, dark = far); every model shown emits this convention natively. The four reference samples are deliberately shared with the sister repos Jens-Duttke/Depth-Anything-3-MONO-ONNX and Jens-Duttke/DepthPro-ONNX-HighPerf so cross-model comparisons are direct.
Click any render to view it full size. The comparison models were run at their own native resolutions with their own preprocessing; the reference images are 1:1, which isolates resolution and says nothing about aspect-ratio behaviour.
Three things worth looking for rather than taking on trust:
- Lotus resolves thin structures (wires, twigs, stems) unusually well for its speed class.
- It has no sky-segmentation head, so skies are whatever the model predicts rather than an explicit mask - compare the upper regions against the Depth Anything V3 Mono renders, which do have one.
- At 256 and 384 the VAE decoder leaves a fine-grained noise texture in flat regions, clearly visible in the foliage sample. It fades at 512 and is gone by 768. If your downstream step amplifies local gradients, this is the reason to prefer 512 over 384 even where the frame rate would allow the smaller tier.
Differences from the upstream pipeline
Two deliberate deviations, both measured:
- The VAE posterior is taken at its mode, not sampled. Upstream calls
.latent_dist.sample(), which re-introduces per-frame randomness into a model chosen for determinism. Measured difference across all reference images and resolutions: mean 0.0001-0.0005 of the[0, 1]readout, max 0.034 - below the FP16 noise floor of the rest of the chain. - Fixed square input instead of aspect-preserving resize. Upstream scales the longest edge to
processing_resand keeps the aspect ratio. This graph takes whatever square you feed it, which is how the consuming application drives every other depth model it ships.
Everything else follows the upstream LotusDPipeline exactly, including the x / 2 + 0.5 denormalization and the mean over the decoder's three output channels.
License
This ONNX build is licensed under the Apache License 2.0; the underlying weights inherit the upstream Lotus license, which is also Apache 2.0. Commercial use, product integration, and service deployment are permitted. This repository is an independent ONNX redistribution and is not affiliated with or endorsed by the upstream Lotus authors.
Acknowledgements
- Lotus by Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu and Ying-Cong Chen - the upstream model and weights (paper, code).
- Stability AI - Stable Diffusion 2.1, which Lotus is built on.
- The Oku3D Media Player - production consumer of this model.
@article{he2024lotus,
title={Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction},
author={He, Jing and Li, Haodong and Yin, Wei and Liang, Yixun and Li, Leheng and Zhou, Kaiqiang and Zhang, Hongbo and Liu, Bingbing and Chen, Ying-Cong},
journal={arXiv preprint arXiv:2409.18124},
year={2024}
}
- Task
- Depth Estimation
- Library
- onnxruntime
- Type
- depth-estimation
- License
- apache-2.0
- Created
- Aug 8, 2026
- Updated
- Aug 21, 2026