rskill-moveit-multi-look_at

rskill-moveit-multi-look_at

Aim a robot-mounted camera at a 3-D point (renamed from openral/rskill-look-at as part of the MoveIt goal-builder rename).

A kind: ros_action rSkill wrapping moveit_msgs/action/MoveGroup — like rskill-moveit-multi-joints, but the goal is a look_at block instead of raw constraints, and ros_integration.goal_builder: look_at selects the LookAtRskill adapter that lowers it into MoveGroup pose constraints.

What this skill does

Given a target point and a camera name, plans a collision-aware arm motion that points the named camera's optical axis at the target — so a later perception query (locate_in_view) or manipulation skill sees the object framed. It is the "look" rung of the recall → navigate → look → verify → manipulate ladder.

How it works

ros_integration.goal_builder: look_at selects LookAtRskill (openral_rskill.look_at_rskill), which lowers the look_at block at dispatch time:

1. Resolve the camera named by look_at.camera (default "wrist") from the host RobotDescription.sensors. No such sensor → ROSConfigError listing the robot's available sensors — never a silent guess. A sensor whose frame_id is itself a robot link (franka's LIBERO eye-in-hand on panda_hand) is constrained directly; a sensor with parent_frame + static_transform_xyz_rpy (so101-style mount) constrains the parent link through the declared offset.
2. Read the camera's current pose over TF2 (the only source of frames) in the goal frame.
3. Place the camera goal — in place (pure re-aim, the default) or at look_at.standoff_m from the target along the current line of approach.
4. Orient it with compute_gaze_pose (ROS optical convention: camera +Z hits look_at.target_xyz; roll about the optical axis left free at tolerance π for planner reachability) and submit MoveGroup position_constraints + orientation_constraints.

plan_only: true, deliberately: OpenRAL's actuation path is the per-waypoint replay through /openral/candidate_action (chunk_size: 1, so the safety supervisor's per-joint envelope check sees every aiming step, and the HAL actuates). Letting move_group also execute on its own controllers would bypass the kernel and double-drive the arm.

Observation → action contract

Input is the goal_params_json look_at block; output is a joint trajectory replayed one waypoint per step() as a 1-row JOINT_POSITION Action chunk.

{"look_at": {"target_xyz": [0.5, 0.0, 0.2], "camera": "wrist"}}

Planner settings (request.group_name, scaling, attempts) are inherited from default_goal_json. Omit standoff_m to re-aim in place; set it to also move the camera to that distance from the target.

GPU-accelerated planning (cuMotion)

On a host that clears the cuMotion GPU floor (RobotCapabilities.supports_cumotion() — Ampere+, CUDA ≥ 13, ~8 GB VRAM), the runner sets MotionPlanRequest.pipeline_id = "isaac_ros_cumotion" so MoveIt plans with NVIDIA's CUDA-accelerated cuMotion pipeline; otherwise it falls back to OMPL. Transparent — same skill, no manifest change — and it never bypasses the safety kernel: the planned trajectory still replays through /openral/candidate_action and is validated waypoint-by-waypoint. Install: see docs/contributing/toolchain.md → "GPU motion planning — cuMotion".

How it was trained / Upstream provenance

Nothing is trained — this rSkill wraps the upstream MoveIt motion planner and computes the gaze pose analytically.

Supported robots / embodiments

A robot must declare the named camera (default wrist) in its robot.yaml sensors, or the skill fails at configure with the available-sensor list. Listed embodiment_tags only gate palette visibility; actual resolution needs move_group up for that robot.

Sensors required / Observation contract

This skill consumes no camera frames through OpenRAL's sensor pipeline — it aims a camera, it doesn't read one. It needs the named camera's frame to exist in TF (declared in the robot manifest + published by robot_state_publisher), plus MoveIt's own subscriptions:

Manifest summary

Full schema: openral_core.schemas.RSkillManifest.

Quick start

from openral_rskill.loader import rSkill
pkg = rSkill.from_yaml("rskills/rskill-moveit-look-at/rskill.yaml")
print(pkg.manifest.name, pkg.manifest.ros_integration.goal_builder)

End-to-end, with a real MoveIt launch up:

# 1. Bring up MoveIt for your robot (example: Panda)
ros2 launch moveit_resources_panda_moveit_config demo.launch.py

# 2. Dispatch a look-at goal (aim the wrist camera at a tabletop point):
ros2 action send_goal /openral/execute_rskill openral_msgs/action/ExecuteRskill \
    "{rskill_id: 'OpenRAL/rskill-moveit-multi-look_at', deadline_s: 30.0, prompt: 'look at the mug',
      goal_params_json: '{\"look_at\": {\"target_xyz\": [0.5, 0.0, 0.2], \"camera\": \"wrist\"}}'}"

Limitations / Roadmap

• Reachability is the planner's call. Roll about the optical axis is left free, but a target outside the arm's dexterous workspace simply fails to plan — there's no base-repositioning fallback here (that's the navigate rung of the ladder).
• Single-camera aim. One camera per dispatch; multi-camera coverage is a reasoner-level concern.
• No velocity / jerk bound at the supervisor. Same posture as rskill-moveit-multi-joints: the per-joint position envelope runs per waypoint; richer bounds are tracked separately.

License

The rSkill package itself (this manifest + README) is Apache-2.0. The wrapped MoveIt code (moveit_msgs IDL, moveit2 planners) is BSD-3-Clause and is installed via ros-${ROS_DISTRO}-moveit, outside this repository. Both postures are commercial-use-permissive.

See also

• MoveIt goal-builder library + rskill-moveit-* rename
• look_at skill + grid-refined approach
• openral_rskill.look_at_rskill — adapter source
• openral_world_state.geometry — gaze math
• rskills/rskill-moveit-joints/ — sibling joint-space MoveIt wrapper
• rskills/rskill-moveit-eef-pose/ — sibling Cartesian end-effector pose wrapper