haplosr-mv

HaploSR-MV — posed multi-view 3D reconstruction

HaploSR-MV turns several posed photos of one object into a single 3D mesh that is measurably closer to ground truth than a single-image reconstruction. It wraps a frozen TripoSR (MIT) with a differentiable multi-view refinement stage: the triplane scene code is optimized so its rendered silhouettes match every view, while an anchor regularizer preserves the sharp detail of the seed view. On device the camera poses come for free from ARKit, which removes the pose-free ambiguity that makes multi-view reconstruction hard.

Base model TripoSR is MIT-licensed, so HaploSR-MV has no commercial restrictions — no attribution requirements, no revenue caps. It's built entirely on permissively-licensed components.

Single photo in, 3D mesh out

HaploSR-MV seeds from a single-image TripoSR reconstruction, then refines it with additional posed views. Here's what the single-photo seed already produces on device (multi-view refinement then sharpens it — see the +33% gate below):

<p align="center"><img src="assets/pair_coke.png" width="680" /></p> <p align="center"><img src="assets/pair_dog.png" width="680" /></p> <p align="center"><img src="assets/pair_avocado.png" width="680" /></p>

Validated result (gate)

Measured on a 4-object subset with elevation-diverse views, ICP-aligned scale-free metrics (Chamfer ↓ better, F-score ↑ better):

4/4 objects improved on both metrics; no regressions.

The one capture rule that matters

Views must be elevation-diverse, not just an azimuthal orbit. Azimuth-only silhouettes carve a generalized cylinder and cannot constrain top/bottom geometry — in early testing that regressed F-score. Guide the capture UX to vary tilt as well as angle (a natural handheld arc around the object does this).

Usage

from haplosr_mv.pipeline import HaploSRMV, RefineConfig, View
from PIL import Image

pipe = HaploSRMV(RefineConfig(steps=150, mc_resolution=256))
views = [
    View(image=Image.open("v0.png"), azimuth_deg=0,   elevation_deg=0),   # seed
    View(image=Image.open("v1.png"), azimuth_deg=55,  elevation_deg=28),
    View(image=Image.open("v2.png"), azimuth_deg=125, elevation_deg=-22),
    # ... ARKit path: pass c2w=<4x4 numpy> instead of azimuth/elevation
]
mesh, _ = pipe.run(views)
mesh.export("out.glb")

Or the CLI: python run.py --demo (A/B on a bundled object), or python run.py --views-dir <dir> with a poses.json.

How it works

1. Seed — TripoSR encodes the canonical view (az=0, el=0) into a triplane.
2. Refine — with the network frozen, Adam optimizes the triplane against a stochastic silhouette loss over all posed views (+ optional photometric), plus an anchor term ‖code − seed‖² (weight 2e-3) that keeps seed detail. 150 steps, lr 8.0 (the triplane std is ~377, so it needs a large lr), 1500 rays/view at 256².
3. Extract — marching cubes on the refined triplane.

Camera convention exactly matches TripoSR's get_spherical_cameras (x-back, y-right, z-up; seed at az=0 el=0), so ARKit c2w poses drop in directly.

Limitations

• The refiner is an optimization loop (~seconds–minutes on Apple GPU/MPS), best run server-side or as a "premium quality" path — not real-time. For real-time on-device fusion use the feed-forward sf3d_mv_encoder.onnx.
• Silhouette-dominant loss: needs reasonable foreground masks per view (alpha, or a segmenter such as BiRefNet/rembg).
• Inherits TripoSR's single-view seed quality as a starting point.

License

MIT (this refinement method + code). Base weights: TripoSR, MIT (stabilityai/TripoSR). No attribution or revenue restrictions.