Reward_Rush_SAC_Mountain_Car

Reward Rush: Mountain Car Continuous SAC

This repository contains a Soft Actor-Critic (SAC) agent trained for the MountainCarContinuous-v0 environment.

Model Architecture

The model uses a multi-layer perceptron actor with the following specifications:

• Input: 2 state observations (Position, Velocity)
• Output: 1 continuous action (Force)
• Architecture:
  • Linear(2, 256) -> ReLU
  • Linear(256, 256) -> ReLU
  • Linear(256, 2)

Note: The final layer outputs two values representing the mean and log standard deviation. For inference, only the first value (mean) is used, passed through a tanh activation function to bound the action to the [-1, 1] range.

Common Implementation Mistakes to Avoid

1. Layer Naming: The state dictionary uses the prefix "net". Architecture definitions must use self.net as an nn.Sequential block to match the keys net.0, net.2, and net.4.
2. Output Dimensions: SAC actors are often saved with a final dimension of 2. Initializing the model with an output dimension of 1 will result in a size mismatch error.
3. Continuous Actions: Mountain Car Continuous requires a numpy array for actions. The tensor output must be flattened and converted to a numpy array before environment steps.
4. Activation: Tanh activation must be applied to the mean output to ensure the action stays within the valid range of [-1, 1].

Download and Test Code

import torch
import torch.nn as nn
import gymnasium as gym
import numpy as np
from huggingface_hub import hf_hub_download

class SACPolicy(nn.Module):
    def __init__(self):
        super(SACPolicy, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(2, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, 2)
        )
        
    def forward(self, x):
        out = self.net(x)
        return torch.tanh(out[:, 0:1])

def run_mountain_car_test():
    path = hf_hub_download(repo_id="Nharen/Reward_Rush_SAC_Mountain_Car", filename="Mountain_car.pth")
    
    model = SACPolicy()
    state_dict = torch.load(path, map_location='cpu', weights_only=True)
    
    if isinstance(state_dict, dict) and "actor_state_dict" in state_dict:
        state_dict = state_dict["actor_state_dict"]
    elif isinstance(state_dict, dict) and "state_dict" in state_dict:
        state_dict = state_dict["state_dict"]

    model.load_state_dict(state_dict)
    model.eval()

    env = gym.make("MountainCarContinuous-v0")
    total_rewards = []

    for _ in range(100):
        state, _ = env.reset()
        episode_reward = 0
        done = False
        while not done:
            state_t = torch.as_tensor(state, dtype=torch.float32).unsqueeze(0)
            with torch.no_grad():
                action = model(state_t).cpu().numpy().flatten()
            
            state, reward, terminated, truncated, _ = env.step(action)
            episode_reward += reward
            done = terminated or truncated
        total_rewards.append(episode_reward)

    print(f"Average Reward: {np.mean(total_rewards)}")
    env.close()

if __name__ == "__main__":
    run_mountain_car_test()