sac-her-PandaPickAndPlace-v3

SAC + HER agent playing PandaPickAndPlace-v3

This is a SAC (Soft Actor-Critic) agent trained with Hindsight Experience Replay (HER) on the PandaPickAndPlace-v3 environment from panda-gym, using stable-baselines3.

Why HER?

PandaPickAndPlace-v3 is a sparse-reward, goal-conditioned task: the agent gets -1 reward every step until the object reaches the target position, 0 otherwise. Plain off-policy RL (SAC/TD3/DDPG) essentially never learns here, because a random policy almost never reaches the goal, so almost every trajectory carries the same uninformative reward signal.

HER fixes this by relabeling failed trajectories: for a rollout that failed to reach its intended goal g but happened to move the object to some other position g', HER stores an extra transition where g' is treated as if it had been the goal all along, with the reward recomputed accordingly. This turns every episode into a source of positive learning signal, "success at a different goal", even when the original goal wasn't reached.

Results

Evaluated over multiple episodes with deterministic actions:

Mean reward = -20.80 +/- 19.18

(episode length capped at 50 steps, -50 = never reached goal, values closer to 0 = faster successful pick-and-place)

Demo

Training hyperparameters

learning_starts must exceed the environment's max episode length (50 steps) so that HER always has at least one complete episode to relabel before the first gradient update — otherwise SB3 raises RuntimeError: Unable to sample before the end of the first episode.

Full training script: train.py.

Usage (with Stable-Baselines3)

pip install stable-baselines3 sb3-contrib huggingface_sb3 panda-gym

import gymnasium as gym
import panda_gym
from huggingface_sb3 import load_from_hub
from stable_baselines3 import SAC
from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize

repo_id = "monshinawatra/sac-her-PandaPickAndPlace-v3"

checkpoint = load_from_hub(repo_id, "sac-her-PandaPickAndPlace-v3.zip")
stats_path = load_from_hub(repo_id, "vec_normalize.pkl")

env = DummyVecEnv([lambda: gym.make("PandaPickAndPlace-v3", render_mode="human")])
env = VecNormalize.load(stats_path, env)
env.training = False
env.norm_reward = False

model = SAC.load(checkpoint, env=env)

obs = env.reset()
for _ in range(1000):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)
    env.render()

Full inference script (with evaluation loop): inference.py.

Environment

• Env ID: PandaPickAndPlace-v3
• Observation space: Dict (observation, achieved_goal, desired_goal) — requires MultiInputPolicy
• Action space: Box(4,) — 3D end-effector displacement + gripper
• Max episode steps: 50