dqn-LunarLander-v3-scratch

DQN (from scratch) - LunarLander-v3

A Deep Q-Network agent for LunarLander-v3, implemented from scratch in PyTorch. The replay buffer, the target network and the Bellman update are written by hand, without an RL library.

This is the educational half of a two-repo pair. The tuned library baseline lives at arabellako22/dqn-LunarLander-v3-sb3.

Results

Both agents were trained for 300,000 environment steps with identical hyperparameters, and evaluated on the same 50 episodes (seeds 100-149, greedy action selection). A score above 200 is generally considered "solved".

Note that the reward distribution is bimodal: most episodes land cleanly above 220, while a minority hover above the pad until the 1000-step time limit and score well below 100. The mean alone is therefore sensitive to the choice of evaluation seeds, which is why the median and the success rate are reported alongside it.

Model description

• Algorithm: vanilla DQN (experience replay + periodically synced target network)
• Network: MLP 8 -> 256 -> 256 -> 4, ReLU
• Observation space: Box(8,) - position, velocity, angle, angular velocity, two leg-contact flags
• Action space: Discrete(4) - noop, left engine, main engine, right engine
• Loss: Huber (smooth L1) on the TD error
• Optimizer: Adam, gradient norm clipped at 10.0

Terminal and truncated transitions are stored separately: bootstrapping is cut only on terminated, so time-limit truncation does not bias value estimates downwards.

Files

Usage

import torch
from huggingface_hub import hf_hub_download

# 1) network definition and weights are both in this repo
model_py = hf_hub_download("arabellako22/dqn-LunarLander-v3-scratch", "model.py")
weights = hf_hub_download("arabellako22/dqn-LunarLander-v3-scratch", "dqn_lunarlander_scratch.pt")

import importlib.util
spec = importlib.util.spec_from_file_location("model", model_py)
model = importlib.util.module_from_spec(spec)
spec.loader.exec_module(model)

q_net = model.QNetwork(obs_dim=8, n_actions=4, hidden=(256, 256))
q_net.load_state_dict(torch.load(weights, map_location="cpu"))
q_net.eval()

Then run a full evaluation:

import gymnasium as gym
import numpy as np
import torch

env = gym.make("LunarLander-v3")
scores = []
for ep in range(20):
    obs, _ = env.reset(seed=1000 + ep)
    total, done = 0.0, False
    while not done:
        with torch.no_grad():
            t = torch.as_tensor(obs, dtype=torch.float32).unsqueeze(0)
            action = int(q_net(t).argmax(dim=1).item())
        obs, reward, terminated, truncated, _ = env.step(action)
        total += reward
        done = terminated or truncated
    scores.append(total)
env.close()
print(f"mean_reward = {np.mean(scores):.1f} +/- {np.std(scores):.1f}")

Training hyperparameters

{
  "env_id": "LunarLander-v3",
  "algorithm": "DQN (from scratch, PyTorch)",
  "obs_dim": 8,
  "n_actions": 4,
  "net_arch": [
    256,
    256
  ],
  "gamma": 0.99,
  "learning_rate": 0.00063,
  "buffer_size": 50000,
  "batch_size": 128,
  "learning_starts": 1000,
  "train_freq": 4,
  "target_update_interval": 250,
  "exploration_fraction": 0.12,
  "exploration_final_eps": 0.05,
  "total_timesteps": 300000,
  "seed": 42
}

Limitations

DQN only supports discrete action spaces, so these weights do not transfer to LunarLanderContinuous-v3 or to continuous-control robotics tasks; use SAC, TD3 or PPO for those. This is plain DQN without Double DQN, dueling heads or prioritized replay, so Q-values are over-estimated and across-seed variance is higher than a tuned library implementation.

References

• Mnih et al., Human-level control through deep reinforcement learning, Nature 2015
• Gymnasium documentation