SB3-PPO-PhoenixNoFrameskip-v4-50M

PPO Agent for Atari PhoenixNoFrameskip-v4 (50M timesteps)

This repository hosts a PPO agent trained on PhoenixNoFrameskip-v4 for 50 million timesteps using Stable-Baselines3.

---

Repository Contents

.
├── train_phoenix.py        # Training script
├── evaluate_phoenix.py     # Evaluation script
├── requirements.txt        # Dependencies
├── LICENSE                 # MIT License
├── README.md
├── model_card.md
├── checkpoints/            # Trained model files (all steps)
├── config/
│ └── hyperparams.json
├── assets/
│ ├── charts/               # Training charts: reward, policy/loss, value, entropy
│ └── eval_video.mp4        # Demo video of trained agent
└── tensorboard/

---

Model Checkpoints

Saved every 5M timesteps:

---

Hyperparameters

See config/hyperparams.json. The training script uses these values:

{
  "learning_rate": "2.5e-4 * linear_schedule",
  "n_steps": 2048,
  "batch_size": 256,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "clip_range": 0.2,
  "ent_coef": 0.005,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "frame_stack": 4,
  "use_sde": false
}

---

Training

You can train the PPO agent from scratch or continue training from an existing checkpoint using train_phoenix.py.

Run:

pip install -r requirements.txt
python train_phoenix.py

Key points about training:

• Uses Stable-Baselines3 PPO with CnnPolicy.
• Environment: PhoenixNoFrameskip-v4 (vectorized with VecFrameStack for 4 frames).
• Default device: GPU if available (torch.cuda), otherwise CPU.
• Total target timesteps: 50 million.
• Training checkpoints are automatically saved every 500,000 steps in SAVE_PATH (ppo_phoenix_model by default).
• If training is interrupted (e.g., keyboard interrupt), the model and training metadata are saved automatically every 100,000 steps via SaveOnInterruptCallback.
• Hyperparameters are defined in the script and partially mirrored in config/hyperparams.json.
• TensorBoard logs are written to ./tensorboard/ for monitoring metrics like reward, policy loss, value loss, and entropy.

---

Training Hardware

The model was trained on:

• CPU: Intel Core i5-13600KF
• GPU: NVIDIA GeForce RTX 3060 12GB
• RAM: 32 GB DDR5 6000MHz CL30
• Storage: NVMe SSD
• OS: openSUSE Tumbleweed

---

Training Charts

Located in assets/charts/:

• ppo_ep_rew_mean.png - Mean episode reward
• ppo_policy_loss.png - Policy loss
• ppo_value_loss.png - Value loss
• ppo_entropy_loss.png - Entropy loss
• ppo_total_loss.png - Total loss

---

Evaluation / Inference

To evaluate the trained agent, use evaluate_phoenix.py. This script allows you to load any checkpoint and run it for multiple episodes.

Example usage:

python evaluate_phoenix.py \
    --model_path checkpoints/ppo_phoenix_50000000_steps.zip \
    --num_episodes 20 \
    --render

Options:

• --model_path: Path to a trained model checkpoint (default: checkpoints/ppo_phoenix_50000000_steps.zip).

• --num_episodes: Number of episodes to evaluate (default: 20).

• --render: Enable rendering of the environment in human mode.

• --use_monitor: Wrap the environment with Gym Monitor to record episode metrics.

Features:

• Handles both Gymnasium and Gym outputs.

• Returns:

  • Mean and standard deviation of rewards over all episodes.
  • Mean and standard deviation of episode lengths.

• Can be used with all checkpoints to compare performance over training.

Minimal Python usage example:

from stable_baselines3 import PPO
import gymnasium as gym

from evaluate_phoenix import make_eval_env, evaluate_model

env = make_eval_env(render=False, use_monitor=True)
model = PPO.load("checkpoints/ppo_phoenix_25000000_steps.zip", env)

mean_reward, std_reward, mean_length, std_length = evaluate_model(model, env, num_episodes=20)

---

Demo Video

Short agent demo: assets/eval_video.mp4

---

License

MIT License (see LICENSE). Atari Phoenix game ROMs are not included. Install via gymnasium[atari] or ale-py.

---

Contact

Maintainer: Aleksandar Nikoloski

Hugging Face: @epsill0n

---