LunarLander-v3-SAC

SAC Agent for LunarLander-v3

This is a trained Soft Actor-Critic (SAC) agent for the LunarLander-v3 environment from Gymnasium.

Model Description

This model was trained using the SAC algorithm with continuous action space on the LunarLander-v3 environment. The agent learns to safely land a lunar module on a landing pad using continuous thrust controls.

Algorithm: Soft Actor-Critic (SAC)

SAC is an off-policy actor-critic algorithm based on the maximum entropy reinforcement learning framework. Key features:

• Entropy Regularization: Encourages exploration by maximizing both expected reward and entropy
• Twin Q-Networks: Uses two critic networks to reduce overestimation bias
• Soft Policy Updates: Smooth target network updates using Polyak averaging

Training Hyperparameters

Environment Details

• Environment: LunarLander-v3
• Action Space: Continuous (Box)
  • Main engine: -1.0 to 1.0
  • Side engines: -1.0 to 1.0
• Observation Space: 8-dimensional vector
  • Position (x, y)
  • Velocity (vx, vy)
  • Angle and angular velocity
  • Leg contact indicators
• Reward: +100-140 for landing on pad, -100 for crash, fuel usage penalty

Model Architecture

Actor Network (Policy)

Input (8) → Linear(256) → ReLU → Linear(256) → ReLU → [Mean(2), Log_Std(2)]

• Outputs mean and log standard deviation for Gaussian policy
• Actions squashed through tanh and scaled by max_action

Critic Network (Twin Q-Networks)

Input (8 + 2) → Linear(256) → ReLU → Linear(256) → ReLU → Q-value(1)

• Two separate Q-networks (Q1, Q2) for double Q-learning
• Takes state-action pairs as input

Files Included

Usage

Installation

pip install gymnasium[box2d] torch numpy

Loading and Using the Model

import torch
import numpy as np
import gymnasium as gym

# Define the Actor Network
class ActorNetwork(torch.nn.Module):
    def __init__(self, state_dim, action_dim, max_action):
        super(ActorNetwork, self).__init__()
        self.l1 = torch.nn.Linear(state_dim, 256)
        self.l2 = torch.nn.Linear(256, 256)
        self.mean = torch.nn.Linear(256, action_dim)
        self.log_std_layer = torch.nn.Linear(256, action_dim)
        self.max_action = float(max_action)

    def forward(self, state):
        x = torch.nn.functional.relu(self.l1(state))
        x = torch.nn.functional.relu(self.l2(x))
        mean = self.mean(x)
        log_std = self.log_std_layer(x)
        log_std = torch.clamp(log_std, -20.0, 2.0)
        return mean, log_std

# Download and load the model
from huggingface_hub import hf_hub_download

# Download model file
actor_path = hf_hub_download(
    repo_id="MohamedMaher003/LunarLander-v3-SAC",
    filename="sac_model_actor.pth"
)

# Initialize environment and model
env = gym.make("LunarLander-v3", continuous=True, render_mode="human")
state_dim = 8
action_dim = 2
max_action = 1.0

# Load actor
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
actor = ActorNetwork(state_dim, action_dim, max_action).to(device)
actor.load_state_dict(torch.load(actor_path, map_location=device))
actor.eval()

# Run evaluation
def select_action(state, actor, device):
    with torch.no_grad():
        state_tensor = torch.FloatTensor(state).unsqueeze(0).to(device)
        mean, _ = actor(state_tensor)
        action = torch.tanh(mean) * actor.max_action
        return action.cpu().numpy().flatten()

state, _ = env.reset()
done = False
total_reward = 0

while not done:
    action = select_action(state, actor, device)
    state, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    done = terminated or truncated

print(f"Episode Reward: {total_reward:.2f}")
env.close()

Training Code

The full training implementation is available at: GitHub Repository

Training was done using:

• Framework: PyTorch
• Logging: Weights & Biases (wandb)

Evaluation Results

The trained agent achieves:

• Average Reward: ~250+ over 10 evaluation episodes
• Success Rate: High landing success rate on the pad
• Stability: Consistent performance across episodes

Citation

If you use this model, please cite:

@misc{maher2024sac-lunarlander,
  author = {Mohamed Maher},
  title = {SAC Agent for LunarLander-v3},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/MohamedMaher003/LunarLander-v3-SAC}}
}

References

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep RL - Haarnoja et al., 2018
• Gymnasium Documentation
• LunarLander Environment

Author

Mohamed Maher - Hugging Face Profile

---

This model was trained as part of a Reinforcement Learning course assignment on State-of-the-Art Model-Free Algorithms at cairo university faculty of engineering.