rskill-sam2_1-any-grasped_object_mask-bf16

rskill-sam2_1-any-grasped_object_mask-bf16

OpenRAL rSkill — SAM 2.1 (Hiera-small) packaged as an Apache-2.0 promptable segmenter (kind: segmenter). It answers one geometric question — "which pixels of this wrist frame belong to the object in the jaws?" — so the HAL's vision attachment-evidence producer can bound a grasped payload from real depth instead of simulator ground truth. One shot per attach / detach / regrasp event. No actuators.

This package wraps hf://facebook/sam2.1-hiera-small with a rskill.yaml manifest that adds capability checking, license surfacing, latency budgets, and local registry integration. It does not copy model weights.

Preview

Real model output on two real in-tree frames (not synthetic), rendered by the same CPU inference path the tests run. Yellow ring = the point prompt; tint = the model's own best-scoring mask.

The right-hand frame is why nothing downstream is allowed to trust the model's confidence. See Quality probe.

What this skill does

Given a camera frame and a point in it, returns the binary mask of whatever object that point lands on. There is no label vocabulary and no class: the skill never says what the pixels are, only which pixels belong to the prompted thing.

Its consumer is the HAL's vision attachment-evidence producer. At a grasp event the producer projects the tool center point into the wrist camera, asks this skill for the mask, intersects the mask with the same frame's depth channel, and fits the ≤16 bounded primitives of an AttachedCollisionObject so the safety kernel plans around the carried payload. It replaces MuJoCo introspection, which gives perfect answers in simulation and does not exist on real hardware.

How it works

SAM 2.1 is a first-class transformers architecture (Sam2Model + Sam2Processor), so it runs in-process via the Sam2Segmenter backend (engine: sam2_hf) — no sidecar venv, no ZMQ, no quantization beyond the bf16 compute dtype. The HAL stays torch-free and reaches it over the openral_msgs/srv/SegmentInView service.

The prompt crosses that service boundary as a 3-D point, not a pixel, so camera intrinsics stay entirely on the perception side. The masks come back as raw mono8 sensor_msgs/Image[] — see the .srv header for why raw and not run-length encoded.

multimask: true returns SAM 2's three nested hypotheses (subpart / part / whole), and all of them cross the service boundary. This skill deliberately does not pick between them: it has no depth, and only geometry — containment between the jaws, payload extent, depth validity — can say which hypothesis is the payload. The backend orders them by area ascending and the HAL producer, which holds the wrist depth frame, selects among them.

Observation → mask contract

Measured numbers

Measured on an RTX 4070 Laptop 8 GB (transformers 5.5.4, torch 2.9.1+cu128, bf16, 512² input, ~3.5 GiB already resident from a co-loaded VLA), by torch.cuda.max_memory_allocated deltas against a pre-load baseline:

Latency is flat across input resolution (46–52 ms from 224² to 1024²) because Sam2Processor always resizes to 1024² internally. Wrist camera resolution is not a lever for this model.

At 297 MiB the segmenter co-resides trivially with a ~3.5 GiB VLA and a simulator on an 8 GB card, so it is held resident and warmed at node activate rather than loaded on demand — the 742 ms first call would not fit inside the HAL's ~100 ms deferred-ack barrier, but the 53 ms warm call fits with room to spare.

Quality probe — and the result that drives the safety design

Two real in-tree frames, prompted with a point at (0.52·W, 0.68·H) (a stand-in for the projected TCP):

The second row is the load-bearing finding for this whole package. A mis-aimed or out-of-context point prompt returned a mask covering most of the image at the model's highest confidence. The model's own score cannot detect that it is wrong.

Therefore every fail-closed gate in the consumer is geometric — containment between the jaws, a payload extent cap, a depth-validity fraction — and never the mask score. mask_scores_advisory is named to say so. Both frames are regression-tested in tests/unit/test_vision_attachment_evidence.py.

Upstream model and training

A thin wrapper around the upstream Apache-2.0 SAM 2.1 checkpoint; weights live upstream and are not copied here.

The checkpoint's config declares sam2_video; loading it into the image-only Sam2Model is the supported subset path and emits a transformers notice to that effect. The image path is all this rSkill uses — there is no video/tracker state, because segmentation is one-shot at an event, not a per-frame track.

Supported robots

Embodiment-agnostic — the only requirement is a wrist RGB camera. The consumer additionally needs registered depth on the same frame, but that is the HAL producer's requirement, not this model's.

Sensors required

Mirrors rskill.yaml::sensors_required.

Manifest summary

Full schema: openral_core.schemas.RSkillManifest.

Quick start

just sync --group sam2   # torch + transformers + pillow for the in-process backend

from openral_core.schemas import RSkillManifest, SegmenterEngine

manifest = RSkillManifest.from_yaml(
    "rskills/rskill-sam2_1-any-grasped_object_mask-bf16/rskill.yaml"
)
assert manifest.segmenter.engine is SegmenterEngine.SAM2_HF

Reproduction

Packaging-only wrapper — no trained numbers to reproduce. The measured latency and VRAM figures above need the reference GPU; the quality probe and the geometric-gate regressions run on CPU:

just sync --group sam2
uv run pytest tests/unit/test_sam2_segmenter.py tests/unit/test_vision_attachment_evidence.py

Evaluation

No benchmarks shipped — packaging-only wrapper; a promptable segmenter has no task success rate. The quality probe above is the honest substitute.

Known limitations

Inherited from the design and not fixed by this skill:

• Articulated objects. Vision gives geometry, not kinematic class. A drawer handle would mask as a free box. The consumer is scoped to free objects.
• Transparent / thin objects. Expected to fail the consumer's depth-validity gate and degrade to the conservative fallback box.
• Self-occlusion. The far side of the object is never observed; only partially mitigated by the consumer's view-ray inflation margin.
• Mass / centre of mass / inertia. Not estimable from vision; they stay None on real hardware, unlike the simulator producer.
• Deformable objects and multi-object grasps. Not addressed — a single positive point means a single object.
• Support contact. This skill establishes what is attached, never what is resting on something.

License

This rSkill package (rskill.yaml, README.md) is apache-2.0. The wrapped weights at hf://facebook/sam2.1-hiera-small are also apache-2.0, so the segmenter is fully commercial-safe (CLAUDE.md §1.9).

See also

• packages/msgs/srv/SegmentInView.srv — the typed HAL↔segmenter boundary, with the mono8-vs-RLE rationale.
• python/hal/src/openral_hal/_vision_attachment_evidence.py — the consumer: geometric gates, primitive fit, and the conservative fallback.
• python/hal/src/openral_hal/_sim_attachment_evidence.py — the simulator counterpart this transfers to real hardware, emitting the identical AttachedCollisionObject contract.
• rskills/omdet-turbo-locator/ — the semantic perception sibling ("find object X"), for contrast.
• CLAUDE.md §6.4 — rSkill packaging contract.