realhand-teleop

RealHand Teleop

RealHand Teleop is an Isaac Sim, Isaac Lab, and Isaac Teleop integration for teleoperating a RealHand P7 bimanual robot equipped with RealHand L6, O6, or L20 dexterous hands from a PICO 4 Ultra headset.

This repository is a robotics software and asset bundle. It does not contain pretrained model weights, NVIDIA runtimes, or the external FFG SDK.

Contents

The package contains:

• RealHand P7 bimanual robot assets.
• RealHand L6, O6, and L20 hand URDF and mesh assets.
• PICO 4 Ultra OpenXR controller teleoperation.
• PICO 4 Ultra OpenXR hand-tracking teleoperation.
• FFG / RealForce glove teleoperation for L6, O6, and L20 hands.
• Controller-driven arm poses with FFG-driven hand joints.
• A 3D-printable PICO controller mount for FFG glove setups.
• Pink-based dual-arm inverse kinematics.
• A self-contained inspection and kitting workcell.
• Runtime and offline FFG calibration tools.

Isaac Lab Robot Assets

Versioned USD packages for the P7 robot with dual L6, O6, or L20 hands are available under assets/isaaclab. They contain the complete robot and verified wrist flange assemblies and can be resolved directly by the corresponding isaaclab_assets configurations.

Supported Modes

Select the hand with P7_REALHAND_HAND_MODEL=l6|o6|l20 and the input mode with P7_REALHAND_INPUT_MODE=controller|handtracking|controller_ffg.

All three hands support all three modes. Gravity and hand self-collision are enabled by default for the inspection scene.

On startup and reset, both P7 arms are straight down beside the body, with open hands, fingers down, and hand backs facing the left/right outer sides of the body (palms inward). All 14 arm joints start at zero; mirrored fixed hand mounts set this orientation. This rest pose is separate from the XR coordinate mapping: controller, hand-tracking, and FFG wrist tracking retain their reference frames. The torso uses convex-decomposition collision geometry so that empty space beside the body remains free for the lowered arms. Initial collision cooking can take longer on the first launch; finger collision settings are independent.

Tested Compatibility

See COMPATIBILITY.md for the tested environment.

Prerequisites

Install the workstation, XR, and glove runtimes separately before using this bundle. This repository only contains RealHand-owned integration code, robot assets, hand assets, scene assets, and launch/calibration scripts.

CloudXR, Isaac Sim, Isaac Lab, Isaac Teleop, the PICO runtime/client, NVIDIA drivers, and the FFG SDK are not redistributed in this repository. Their upstream license terms apply independently.

Workspace Paths

The commands below use a configurable workspace instead of a machine-specific path. Replace $HOME/realhand with any local directory you prefer:

export REALHAND_WORKSPACE="$HOME/realhand"
export REALHAND_TELEOP_ROOT="$REALHAND_WORKSPACE/realhand-teleop"
export ISAACLAB_ROOT="$REALHAND_WORKSPACE/IsaacLab"
export ISAACLAB_ENV="$REALHAND_WORKSPACE/env_isaaclab"

FFG SDK Setup

controller and handtracking modes do not need the FFG SDK. Install it only when using P7_REALHAND_INPUT_MODE=controller_ffg.

The launcher looks for the SDK at P7_REALHAND_FFG_SDK_PATH. The recommended layout places the FFG SDK as a sibling of the downloaded RealHand Teleop repo:

$REALHAND_WORKSPACE/FFG_realhand_pure_python-main

Recommended install after creating the Isaac Lab Python environment:

mkdir -p "$REALHAND_WORKSPACE"
cd "$REALHAND_WORKSPACE"

git clone https://github.com/RealHand-Robotics/FFG_realhand_pure_python.git \
  FFG_realhand_pure_python-main

source "$ISAACLAB_ENV/bin/activate"
python -m pip install -e "$REALHAND_WORKSPACE/FFG_realhand_pure_python-main"

export P7_REALHAND_FFG_SDK_PATH="$REALHAND_WORKSPACE/FFG_realhand_pure_python-main"

The upstream FFG SDK also documents a standalone Conda environment. For this Isaac Teleop bundle, install the SDK into the Isaac Lab Python environment or make it visible through P7_REALHAND_FFG_SDK_PATH/PYTHONPATH.

After downloading this repository, verify that the package is importable and that the gloves are visible from the repository root:

python -c "import realhand_pure_python; print('realhand_pure_python ok')"
python scripts/teleop/scan_ffg_gloves.py

If Linux reports permission denied for /dev/ttyUSB*, add the user to dialout, then log out and back in:

sudo usermod -aG dialout "$USER"

Download

Set the workspace variables above, install or invoke the Hugging Face CLI, then download the package:

mkdir -p "$REALHAND_WORKSPACE"
cd "$REALHAND_WORKSPACE"

uvx hf download realhandinc/realhand-teleop \
  --revision v0.2.0 \
  --local-dir realhand-teleop

cd "$REALHAND_TELEOP_ROOT"

Point the launcher to the Isaac Lab checkout and Python environment when they are not in the default locations:

export ISAACLAB_ROOT="$REALHAND_WORKSPACE/IsaacLab"
export ISAACLAB_ENV="$REALHAND_WORKSPACE/env_isaaclab"

Install And Validate

source "$ISAACLAB_ENV/bin/activate"

python -m pip install -e \
  examples/isaac_lab/p7_realhand_bimanual/source

python scripts/teleop/check_install.py
python scripts/teleop/test_retargeters.py

The editable install is convenient but not required by the launcher because it also adds the extension and shared retargeter paths to PYTHONPATH.

First-Time CloudXR Setup

Accept the NVIDIA CloudXR EULA once before the first XR session:

source "$ISAACLAB_ENV/bin/activate"
python -m isaacteleop.cloudxr --accept-eula

After the runtime reports that it is running, press Ctrl+C. The regular launcher starts CloudXR automatically, so a separate CloudXR terminal is not required for normal use.

Run

Activate the environment and enter the downloaded repository:

source "$ISAACLAB_ENV/bin/activate"
cd "$REALHAND_TELEOP_ROOT"

L6 controller mode:

P7_REALHAND_HAND_MODEL=l6 \
P7_REALHAND_INPUT_MODE=controller \
PICO4U_DEVICE=cpu \
bash scripts/teleop/run_pico4u_teleop.sh

O6 hand-tracking mode:

P7_REALHAND_HAND_MODEL=o6 \
P7_REALHAND_INPUT_MODE=handtracking \
PICO4U_DEVICE=cpu \
bash scripts/teleop/run_pico4u_teleop.sh

L20 controller arms plus FFG hands:

P7_REALHAND_HAND_MODEL=l20 \
P7_REALHAND_INPUT_MODE=controller_ffg \
P7_REALHAND_FFG_SDK_PATH="$REALHAND_WORKSPACE/FFG_realhand_pure_python-main" \
PICO4U_DEVICE=cpu \
bash scripts/teleop/run_pico4u_teleop.sh

Use P7_REALHAND_HAND_MODEL=l6, o6, or l20 with any of the three supported input modes.

FFG USB Discovery

The FFG reader auto-discovers USB serial devices and uses the glove-reported side byte (left/right) from the RealForce version query. Set P7_REALHAND_FFG_LEFT_PORT and P7_REALHAND_FFG_RIGHT_PORT only when explicit assignment is needed. Leaving them unset, empty, or set to auto uses discovery.

Check USB access before launching Isaac:

python scripts/teleop/scan_ffg_gloves.py

If Linux reports permission denied for /dev/ttyUSB*, add the user to dialout and log out and back in:

sudo usermod -aG dialout "$USER"

For the current terminal only, use sg dialout -c '<command>' after group membership is updated.

FFG PICO Controller Mount

The package includes a 3D-printable PICO controller holder for FFG glove use:

hardware/ffg_pico_controller_mount/finalpico2.3mf

Print two copies for each hand. Assemble the two printed parts symmetrically around the PICO controller, then mount the assembled holder on the back of the FFG glove. With the controller fixed on the glove back, controller_ffg mode can use the PICO controller pose for the P7 arm while the FFG glove controls the RealHand joints.

FFG Calibration

You can calibrate FFG gloves from the Isaac desktop UI in controller_ffg mode or from the offline script:

P7_REALHAND_HAND_MODEL=l20 \
python scripts/teleop/calibrate_ffg_gloves.py --model l20 --force

The output is written to:

calibration/ffg_calibration_<model>_dual.yml

The launcher automatically uses this file when it exists. To override it:

export P7_REALHAND_FFG_RAW_CALIBRATION_PATH=/path/to/ffg_calibration_l20_dual.yml

L6/O6 Calibration Poses

L6 and O6 use a short five-pose calibration because their thumbs do not need the full L20 opposition workspace.

Open palm:

• Fully open the hand.
• Keep all five fingers naturally straight.
• Keep the thumb relaxed and away from the palm.
• Do not hyperextend or force the fingers backward.

Full fist:

• Curl index, middle, ring, and pinky as far as comfortably possible.
• Curl the thumb naturally across the front of the fingers.
• Hold a normal fist, not a squeezed or strained fist.

Thumb-only full curl:

• Keep index, middle, ring, and pinky open.
• Curl only the thumb inward across the palm.
• Move the thumb to its comfortable closed limit.
• Avoid bending the other fingers while holding this pose.

Thumb to index fingertip:

• Touch the thumb tip lightly to the index fingertip.
• Use fingertip-to-fingertip contact, not thumb pad to finger side.
• Keep middle, ring, and pinky mostly open and relaxed.
• The contact should be gentle; do not squeeze.

Thumb to middle fingertip:

• Touch the thumb tip lightly to the middle fingertip.
• Keep the index, ring, and pinky mostly open.
• Keep the palm relaxed and avoid twisting the wrist to compensate.
• This pose gives L6/O6 a second thumb-opposition reference without asking for unreachable ring or pinky opposition.

L20 Calibration Poses

L20 uses a richer seven-pose calibration because its thumb has a larger lateral workspace.

Open palm:

• Fully open all fingers.
• Keep the palm relaxed.
• Keep the thumb away from the palm in a natural open-hand posture.

Full fist:

• Curl all four fingers fully.
• Curl the thumb naturally across the palm/fingers.
• Use the most comfortable full fist you can repeat consistently.

Thumb-only full curl:

• Keep the four non-thumb fingers open.
• Curl only the thumb across the palm to its comfortable limit.
• This pose teaches thumb curl without letting other fingers influence it.

Thumb to index fingertip:

• Touch thumb tip to index fingertip.
• Keep the other three fingers relaxed and mostly open.

Thumb to middle fingertip:

• Touch thumb tip to middle fingertip.
• Keep index, ring, and pinky relaxed and mostly open.

Thumb to ring fingertip:

• Touch thumb tip to ring fingertip.
• Keep index, middle, and pinky relaxed.
• Let the thumb move laterally; avoid forcing the wrist to create the contact.

Thumb to pinky fingertip:

• Touch thumb tip to pinky fingertip.
• Keep the contact light and repeatable.
• This pose is the far lateral boundary for L20 thumb opposition.

For every pose, hold still during the sampling countdown. If a pose feels ambiguous, prefer a relaxed, repeatable posture over an extreme one.

Default Scene

The bundled inspection_kitting scene contains:

• A compact inspection table.
• Two reachable sorting baskets.
• A thin cube-frame grasp object.
• Collision and rigid-body settings tuned for the P7 and RealHand setup.

The scene is self-contained and does not require external robot assets or an NVIDIA Nucleus asset server.

Project Layout

src/plugins/realforce_glove/
    FFG / RealForce glove input plugin.

src/retargeters/realhand/
    Shared RealHand L6/O6/L20 retargeting code.

examples/teleop/python/config/realhand/
    Retargeting URDF and DexPilot configuration files.

examples/isaac_lab/p7_realhand_bimanual/
    P7 + RealHand assets, Isaac Lab task, scene, scripts, and documentation.

hardware/ffg_pico_controller_mount/
    3D-printable PICO controller holder for FFG glove mounting.

scripts/teleop/
    Stable root launch, FFG calibration, USB scan, and validation entry points.

Useful Runtime Settings

P7_REALHAND_HAND_MODEL=o6
P7_REALHAND_INPUT_MODE=controller_ffg
P7_REALHAND_SCENE_MODE=inspection_kitting
P7_REALHAND_SELF_COLLISION=1
P7_REALHAND_L20_SELF_COLLISION_FILTERS=1
P7_REALHAND_ROBOT_YAW_DEG=90
P7_REALHAND_WORKSPACE_CENTER_Z=0.82
P7_REALHAND_CONTROLLER_POSITION_SMOOTHING_ALPHA=1.0
P7_REALHAND_CONTROLLER_ORIENTATION_SMOOTHING_ALPHA=0.85
P7_REALHAND_CONTROLLER_MAX_ORIENTATION_STEP_DEG=90.0
P7_REALHAND_HAND_CURL_SMOOTHING_ALPHA=0.65
P7_REALHAND_HAND_THUMB_FLEX_SCALE=1.15
P7_REALHAND_HAND_THUMB_CLOSE_GAIN=1.80
P7_REALHAND_HAND_THUMB_OPPOSITION_GAIN=1.80
P7_REALHAND_IK_POSITION_COST=42.0
P7_REALHAND_IK_ORIENTATION_COST=2.0
P7_REALHAND_IK_LM_DAMPING=3.0
P7_REALHAND_IK_GAIN=0.85
P7_REALHAND_FFG_SDK_PATH=$REALHAND_WORKSPACE/FFG_realhand_pure_python-main
P7_REALHAND_FFG_LEFT_PORT=auto
P7_REALHAND_FFG_RIGHT_PORT=auto
P7_REALHAND_FFG_BAUDRATE=0
P7_REALHAND_FFG_RAW_CALIBRATION_PATH=calibration/ffg_calibration_o6_dual.yml

Legacy P7L6_* spellings remain accepted for existing scripts.

More PICO and OpenXR notes are available in examples/isaac_lab/p7_realhand_bimanual/docs/OPENXR_PICO4U.md.

Scope And Safety

This package controls the simulated P7 and RealHand system. It is not a certified real-robot safety controller. Validate workspace limits, collision behavior, emergency stops, and command transport before adapting it to physical hardware.

License

RealHand-owned code, P7/L6/O6/L20 assets, and the inspection scene are released under Apache-2.0. Files derived from Isaac Lab retain their BSD-3-Clause headers and terms. See LICENSE, NOTICE, and LICENSES/ for details.