lunar-lander-v3-dqn

🚀 Dueling Double DQN Agent for LunarLander-v3

This repository provides a high-performance Dueling Double Deep Q-Network (Dueling DDQN) agent trained to master the LunarLander-v3 environment from Gymnasium (Farama Foundation).

---

📊 Benchmark & Evaluation Results

Tested on 100 consecutive evaluation episodes with deterministic evaluation policy ($\epsilon = 0.0$):

🏆 Environment Status: SOLVED (Mean score $\ge 200$ points consistently across diverse initial conditions).

---

🧠 Neural Network & Agent Architecture

The agent implements a Dueling Deep Q-Network combined with Double Q-Learning, allowing decoupled state-value estimation from action-advantage estimation.

                  ┌───────────────────────────────┐
                  │ State Input: s ∈ ℝ⁸           │
                  └───────────────┬───────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Linear(8, 128) + ReLU         │
                  └───────────────┬───────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Linear(128, 128) + ReLU       │
                  └───────────────┬───────────────┘
                                  │
                ┌─────────────────┴─────────────────┐
                │ (Shared Feature Map: ℝ¹²⁸)        │
                ▼                                   ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│ Value Stream V(s)             │   │ Advantage Stream A(s, a)      │
│ Linear(128, 64) + ReLU        │   │ Linear(128, 64) + ReLU        │
│ Linear(64, 1)                 │   │ Linear(64, 4)                 │
└───────────────┬───────────────┘   └───────────────┬───────────────┘
                │                                   │
                └─────────────────┬─────────────────┘
                                  │
  ┌───────────────────────────────▼────────────────────────────────────────┐
  │ Q(s, a) = V(s) + ( A(s, a) - 1/|A| ∑_{a'} A(s, a') )                  │
  └───────────────────────────────┬────────────────────────────────────────┘
                                  │
                  ┌───────────────▼───────────────┐
                  │ Output: Q-values ∈ ℝ⁴         │
                  └───────────────────────────────┘

📐 Layer Specifications

---

⚙️ Hyperparameters

---

🪐 Environment Specifications (LunarLander-v3)

State Space (8 Continuous Dimensions)

1. $x$: Horizontal coordinate
2. $y$: Vertical coordinate
3. $v_x$: Horizontal linear velocity
4. $v_y$: Vertical linear velocity
5. $\theta$: Lander orientation angle (radians)
6. $\omega$: Angular velocity
7. $l_1$: Left leg ground contact flag (0.0 or 1.0)
8. $l_2$: Right leg ground contact flag (0.0 or 1.0)

Action Space (4 Discrete Actions)

• 0: Do nothing (Rest)
• 1: Fire left orientation engine (Tilts right)
• 2: Fire main engine (Thrusts upward)
• 3: Fire right orientation engine (Tilts left)

---

📦 Files Included

• best_model.pth: Best performing checkpoint during training (Mean reward: 273.14).
• latest_model.pth: Final training checkpoint after complete episode run.
• dqn_agent.py: Standalone PyTorch implementation of DuelingQNetwork and DQNAgent.
• evaluate_hf.py: Turn-key evaluation and graphical test script.
• README.md: Comprehensive model card & benchmark report.

---

🎮 Quick Start & Evaluation

1. Installation

pip install torch gymnasium[box2d] huggingface_hub pygame

2. Run Evaluation with Visual Rendering

python evaluate_hf.py --repo moona-ai/lunar-lander-v3-dqn --episodes 5 --render human

3. Run in Challenging Wind Conditions

python evaluate_hf.py --repo moona-ai/lunar-lander-v3-dqn --episodes 5 --render human --wind

---

💻 Direct Python Integration

import gymnasium as gym
import torch
from huggingface_hub import hf_hub_download
from dqn_agent import DQNAgent

# 1. Download best weights directly from Hugging Face Hub
model_path = hf_hub_download(repo_id="moona-ai/lunar-lander-v3-dqn", filename="best_model.pth")

# 2. Instantiate agent & load checkpoint
agent = DQNAgent(state_size=8, action_size=4)
agent.load(model_path)
print("✅ LunarLander Dueling DDQN model loaded successfully!")

# 3. Create Gymnasium environment & run test episode
env = gym.make("LunarLander-v3", render_mode="human")
state, _ = env.reset()
done = False
total_reward = 0.0

while not done:
    # Evaluate with deterministic greedy policy (evaluate=True)
    action, q_values = agent.act(state, evaluate=True)
    state, reward, terminated, truncated, _ = env.step(action)
    total_reward += reward
    done = terminated or truncated

print(f"🎉 Episode Finished! Total Reward: {total_reward:.2f}")
env.close()

---

🏷️ Citation & Acknowledgements

• Environment: Gymnasium Box2D LunarLander
• Algorithm References:
  • Dueling Network Architectures for Deep Reinforcement Learning (Wang et al., 2016)
  • Deep Reinforcement Learning with Double Q-learning (van Hasselt et al., 2015)
• Trained & Uploaded with: LunarLander RL Studio