sam-3d-objects-bf16

SAM 3D Objects — MLX BF16

BF16 weights of Meta SAM 3D Objects converted for the MLX runtime in mlx-vlm (mlx_vlm/models/sam3d_objects), bundled with the MoGe-3 depth estimator (mlx-community/moge-3-vitl-mlx-fp32, float32) in place of the upstream pipeline's MoGe-v1. This repository holds weights and metadata only; the inference code lives in mlx-vlm. This is an unofficial derivative. SAM components retain the SAM License; MoGe-3 retains MIT; FlexiCubes retains Apache-2.0.

The seven safetensors files hold 3,288 BF16 SAM tensors covering the structure and pose flow model, the sparse-latent flow model, image and point-map conditioners, the occupancy decoder, both Gaussian decoders (32 or 4 Gaussians per voxel), and the mesh decoder, plus the 607 float32 MoGe-3 tensors under the depth_model prefix. The model generates geometry and pose; it has no text or tool-calling interface.

Usage

Install an mlx-vlm version that includes sam3d_objects, download this repository, and run the CLI or the Python API:

pip install mlx-vlm
hf download mlx-community/sam-3d-objects-bf16 --local-dir sam-3d-objects-bf16
python -m mlx_vlm.models.sam3d_objects.generate \
  --model sam-3d-objects-bf16 --image image.png --mask mask.png \
  --output output --formats gaussian mesh

from mlx_vlm.models.sam3d_objects.generate import read_image, read_mask
from mlx_vlm.models.sam3d_objects.pipeline import Pipeline

pipeline = Pipeline.from_pretrained("sam-3d-objects-bf16")
image, mask = read_image("image.png")[..., :3], read_mask("mask.png")
result = pipeline.generate(image, mask, seed=42)

pipeline.astream(requests) streams ordered events for an iterable or async iterable of Request values, one request at a time with backpressure; the same weights also load through mlx_vlm.utils.load_model. Inputs are HWC RGB/RGBA MLX arrays and HW masks. Automatic MoGe-3 depth is included, in MoGe-3's metric scale; external HWC point maps use +X left, +Y up, +Z forward with NaN for invalid points. Results include local-space Gaussians, mesh vertices/faces/colors, latent features, occupied voxels, and a separate pose (rotation, translation, scale). The mlx-vlm model README documents the full API, CLI, JSONL streaming, and the inference optimizations.

Conversion and validation

SAM weights were converted directly from the official tensor-only PyTorch ZIP checkpoints through a restricted stdlib reader. Tensor bytes are first preserved in safetensors, then mapped to MLX layouts and cast to BF16. The duplicate occupancy decoder is removed. The MoGe-3 tensors are the mlx-community MLX conversion, bundled unchanged in float32. Conversion and the model runtime import neither PyTorch nor NumPy; the CLI reads images with Pillow and writes PLY/OBJ files with NumPy.

All parameter names/shapes were checked with strict loading, every SAM floating tensor is BF16, and the loaded model has zero trainable parameters. Component checks against the original reference in FP32 MLX CPU cover attention, both flow models, occupancy decoding, interpolation, and FlexiCubes. Maximum absolute error was below 0.000002 on these fixtures; mesh face indices matched exactly. The MoGe-3 port is validated against its PyTorch reference in the moge3 model repository. This is not a claim of bitwise BF16 end-to-end parity or a reconstruction-quality evaluation.

See validation.json and conversion.json for details. The original sources are SAM 3D Objects and MoGe-3.

Local performance

Hardware: Apple M5 Max, 128 GiB unified memory. Input: 448×672 RGB plus object mask (id3_shutterstock_WildAnimal_Waterhole_2010559391/image.png and 0.png resized to 672x448); automatic MoGe-3 depth, 25 structure steps and 25 latent steps, both Gaussian heads and mesh. One warmup was excluded; the same seed produced identical latent outputs.

Two additional async requests matched the synchronous latent output while the event-loop heartbeat continued. These measurements describe one input.