triposr-ios

<div align="center">

TripoSR iOS (ONNX)

Single image → 3D mesh, on your iPhone.

The ONNX-converted encoder from TripoSR by Stability AI × Tripo AI — optimized for on-device inference.

<br/> <table> <tr> <td align="center"><b>419M</b><br/>_Parameters</td> <td align="center"><b>1.6 GB</b><br/>_{Model Size}</td> <td align="center"><b>&lt; 0.5s</b><br/>_{Inference (A100)}</td> <td align="center"><b>ONNX</b><br/>_Format</td> <td align="center"><b>MIT</b><br/>_License</td> </tr> </table> </div>

---

Demo

<table> <tr> <td align="center"><b>Input Photo</b></td> <td align="center"><b>3D Output</b></td> </tr> <tr> <td align="center"><img src="assets/input_photo.jpeg" width="300" /></td> <td align="center"><video src="https://huggingface.co/jc-builds/triposr-ios/resolve/main/assets/3d_output.mp4" controls autoplay loop width="300"></video></td> </tr> </table>

More examples  —  single photo in, 3D mesh out

<p align="center"><i>Each mesh reconstructed on device from a single image.</i></p> <p align="center"><img src="assets/pair_coke.png" width="680" /></p> <p align="center"><img src="assets/pair_dog.png" width="680" /></p> <p align="center"><img src="assets/pair_avocado.png" width="680" /></p>

---

Benchmarks

Evaluated on GSO and OmniObject3D datasets. Results from the TripoSR paper.

F-Score @ 0.1  (higher is better)

Chamfer Distance  (lower is better)

F-Score Across Thresholds

Cross-Dataset Comparison

<details> <summary><b>Full Results Table</b></summary> <br/>

GSO Dataset

OmniObject3D Dataset

</details>

---

Architecture

One forward pass — no diffusion, no iterative denoising.

graph LR
    A["Input Image<br/>(512x512)"] --> B["DINO ViT-B/16<br/>Image Tokenizer"]
    B --> C["Transformer Decoder<br/>+ Cross-Attention"]
    C --> D["Post Processor<br/>Triplane Features"]
    D --> E["Marching Cubes<br/>3D Mesh"]

    style A fill:#4a9eff,stroke:#30363d,color:#fff
    style B fill:#7c3aed,stroke:#30363d,color:#fff
    style C fill:#7c3aed,stroke:#30363d,color:#fff
    style D fill:#7c3aed,stroke:#30363d,color:#fff
    style E fill:#3fb950,stroke:#30363d,color:#fff

---

PyTorch vs. This Model

---

What I Learned Getting This to Work Well

Getting TripoSR to produce clean 3D meshes on a phone took more work than just converting the model to ONNX. The raw model expects a very specific kind of input — a single object, centered, on a neutral background — and if you just feed it a raw photo, the results are pretty rough.

The biggest improvement came from stripping the background before inference. I'm using Apple's Vision framework (VNGenerateForegroundInstanceMaskRequest on iOS 17+) to automatically detect and isolate the main subject. This is the same API that powers the "lift subject from background" feature in Photos — it's fast, runs on-device, and handles edges surprisingly well. The isolated subject gets composited onto a flat gray background (RGB 0.5, 0.5, 0.5), which matches what TripoSR was trained on.

The second big win was smart cropping and centering. After removing the background, I analyze the remaining foreground pixels to find the bounding box, then scale and center the subject so it fills roughly 85-95% of the frame. Too small and the model loses detail; too large and geometry gets clipped. The fill ratio adapts based on the object's shape — tall/narrow objects get a bit more breathing room, compact objects fill more of the frame. A small amount of padding (2-6%) prevents edge artifacts.

I also added a lightweight image enhancement pipeline before inference: noise reduction, luminance sharpening, and edge smoothing after the resize. Lanczos resampling (instead of bilinear) for the 512x512 resize made a noticeable difference in preserving fine detail. All of this runs through Core Image with Metal acceleration, so it adds minimal overhead.

The full pipeline — background removal, crop, center, enhance, infer — runs entirely on-device in Haplo AI. No server, no internet required.

---

Quick Start

<details open> <summary><b>Python</b></summary>

import onnxruntime as ort
import numpy as np
from PIL import Image

session = ort.InferenceSession(
    "triposr_encoder.onnx",
    providers=['CPUExecutionProvider']  # or 'CoreMLExecutionProvider'
)

image = Image.open("photo.png").convert("RGB").resize((512, 512))
input_array = np.array(image).astype(np.float32) / 255.0
input_array = input_array.transpose(2, 0, 1)[np.newaxis, ...]

scene_codes = session.run(None, {"input_image": input_array})[0]
# scene_codes.shape == (1, 3, 40, 64, 64)

</details> <details> <summary><b>Swift (iOS)</b></summary>

import OnnxRuntimeBindings

let session = try ORTSession(env: env, modelPath: modelPath, sessionOptions: nil)

let inputTensor = try ORTValue(
    tensorData: imageData,
    elementType: .float,
    shape: [1, 3, 512, 512]
)

let outputs = try session.run(
    withInputs: ["input_image": inputTensor],
    outputNames: ["scene_codes"]
)

</details>

---

Files

---

Citation

@article{TripoSR2024,
  title={TripoSR: Fast 3D Object Reconstruction from a Single Image},
  author={Tochilkin, Dmitry and Pankratz, David and Liu, Zexiang and Huang, Zixuan
          and Letts, Adam and Li, Yangguang and Liang, Ding and Laforte, Christian
          and Jampani, Varun and Cao, Yan-Pei},
  journal={arXiv preprint arXiv:2403.02151},
  year={2024}
}

<div align="center"> _{MIT License &bull; Based on <a href="https://github.com/VAST-AI-Research/TripoSR">TripoSR</a> by Stability AI &times; Tripo AI} </div>