ppo-pusher-v5-vecnormalize

PPO Pusher-v5 with VecNormalize

This repository contains the best evaluation checkpoint from a Stable-Baselines3 PPO training run on Gymnasium Pusher-v5.

The checkpoint passed both release gates:

1. trained mean reward > random-policy mean reward
2. Deterministic action replay produced the same episode reward in the isolated MuJoCo rendering process.

Final evaluation

For Pusher-v5, rewards are negative; values closer to zero are better.

Training configuration

Included files

Replay

<video controls loop muted playsinline src="replay.mp4"></video>

Open or download replay.mp4

Load the model

from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.vec_env import VecNormalize

env = make_vec_env(
    "Pusher-v5",
    n_envs=1,
)
env = VecNormalize.load(
    "vec_normalize.pkl",
    env,
)
env.training = False
env.norm_reward = False

model = PPO.load(
    "ppo-Pusher-v5.zip",
    env=env,
    device="cpu",
)

obs = env.reset()
for step in range(100):
    action, state = model.predict(
        obs,
        deterministic=True,
    )
    obs, rewards, dones, infos = env.step(action)

env.close()

The VecNormalize statistics must be loaded with the PPO checkpoint. Raw observations will not reproduce the reported result.

Rendering architecture

The evaluation and action-generation process imports Stable-Baselines3 but performs no rendering. A separate MuJoCo-only process replays the deterministic actions using the same seed. This prevents native OpenGL conflicts in headless Colab runtimes.

Package versions

• Gymnasium: 1.3.0
• Stable-Baselines3: 2.9.0
• MuJoCo: 3.12.0
• NumPy: 2.1.3
• PyTorch: 2.11.0+cpu
• ImageIO: 2.37.4

Reusable safe rendering module

The safe_mujoco_render.py module packages the subprocess-isolated evaluation and rendering workflow for other Gymnasium MuJoCo environments.

Supported configuration:

• Stable-Baselines3 PPO checkpoints
• Matching VecNormalize statistics
• Continuous Box action spaces
• EGL with OSMesa fallback
• Optional MP4 output
• Deterministic replay-reward verification

Usage

from safe_mujoco_render import (
    evaluate_and_render_ppo_mujoco,
)

result = evaluate_and_render_ppo_mujoco(
    env_id="Pusher-v5",
    model_path="ppo-Pusher-v5.zip",
    vecnormalize_path="vec_normalize.pkl",
    video_path="replay_from_module.mp4",
    device="cpu",
    n_eval_episodes=30,
    n_random_episodes=30,
    eval_seed=1000,
    random_seed=2000,
    replay_seed=42,
    width=640,
    height=480,
    backends=("egl", "osmesa"),
)

print(result)

Set video_path=None to validate frame generation without writing a video file. The function raises an exception if evaluation and rendering replay rewards do not match.