melee-rl-fox-mango

Melee RL — Fox vs Jigglypuff via Dolphin + OpenEnv

A reinforcement learning system that trains a Fox AI agent to play Super Smash Bros. Melee in real-time. The agent connects to Slippi Dolphin via libmelee and learns through self-play using PPO, served over HTTP with Meta's OpenEnv framework.

How It Works

Training Script (PPO)          OpenEnv Server              Slippi Dolphin
┌──────────────────┐     HTTP/WS      ┌──────────────┐      libmelee      ┌──────────────┐
│ dolphin_train.py │ ──SmashAction──> │ FastAPI app  │ ──controller───> │  Melee game  │
│                  │ <─SmashObs─────  │ (port 8000)  │ <─gamestate────  │  (60 FPS)    │
│  CompetitiveMelee│                  │ EmulatorEnv  │                  │  Fox vs Puff │
│  Reward shaping  │                  │ Server       │                  │  on FD       │
└──────────────────┘                  └──────────────┘                  └──────────────┘

1. OpenEnv server launches Dolphin, navigates menus, and exposes reset() / step() over HTTP
2. Training client sends controller inputs (stick, buttons) each frame and receives game state back
3. Reward shaping (CompetitiveMeleeReward) computes per-frame reward on the client side
4. PPO updates the policy every 2048 frames

Training Pipeline

Phase 1 — Physics Simulator (fast, offline)

Train base policies in a custom Melee physics engine — no emulator needed.

pip install -r requirements.txt

# Train Puff (defensive, spacing-based)
python3 train.py --total-frames 1500000 --device cpu

# Train Fox/Mango (aggressive, pressure-based)
python3 mango_trainer.py --total-frames 500000 --device cpu

Produces checkpoints/puff_final.pt and checkpoints/mango_final.pt.

Phase 2 — Dolphin Fine-Tuning (real game)

Fine-tune the sim-trained Fox against real Melee running in Dolphin.

# Terminal 1: Start the OpenEnv server (launches Dolphin)
cd emulator_env && uv run --project . server

# Terminal 2: Train Fox via PPO against CPU or Puff model
cd emulator_env && uv run python dolphin_train.py --agent mango \
  --checkpoint ../checkpoints/mango_final.pt --total-frames 500000

Configuration

All environment config is in emulator_env/.env:

DOLPHIN_PATH=~/Library/Application Support/Slippi Launcher/netplay
ISO_PATH=~/Downloads/Super Smash Bros. Melee (USA) (En,Ja) (v1.02).iso
P1_CHARACTER=FOX
P2_CHARACTER=JIGGLYPUFF
CPU_LEVEL=7                # 0 = model-driven P2, 1-9 = Dolphin CPU AI
TRAINING_MODE=NORMAL       # NORMAL or RECOVERY
# P2_CHECKPOINT_PATH=../checkpoints/puff_final.pt  # uncomment when CPU_LEVEL=0

Model Architecture

ActorCriticMLP — shared backbone with separate actor and critic heads.

Observation Vector (26-dim)

Per player (13 dims): position (x, y), velocities (x, y), damage (normalized), stocks (normalized), on_ground, facing, action_state, hitstun.

Action Space

MultiDiscrete([5, 4, 2, 2, 2, 2]) mapping to GameCube controller:

Reward System

CompetitiveMeleeReward (rewards/competitive.py) — aggressive Fox / Mango style:

PPO Hyperparameters

Tech Stack

Project Structure

├── physics/                    # Custom Melee physics simulator
│   ├── state.py                #   Game state dataclasses
│   ├── constants.py            #   Character + stage constants
│   ├── melee_physics.py        #   Knockback, hitstun, gravity
│   └── simulator.py            #   Deterministic step function
│
├── envs/                       # RL environments
│   ├── melee_sim_env.py        #   Gymnasium env (physics sim)
│   └── melee_torchrl_env.py    #   TorchRL env (alternative)
│
├── rewards/                    # Reward shaping
│   ├── competitive.py          #   CompetitiveMeleeReward (Fox/Mango)
│   └── puff.py                 #   PuffReward (Jigglypuff)
│
├── emulator_env/               # Dolphin integration (OpenEnv)
│   ├── server/
│   │   ├── app.py              #   FastAPI server (OpenEnv create_app)
│   │   └── emulator_env_environment.py  # Dolphin wrapper (reset/step)
│   ├── client.py               #   EmulatorEnv WebSocket client
│   ├── dolphin_train.py        #   PPO fine-tuning loop
│   ├── policy_runner.py        #   ActorCriticMLP + obs/action conversion
│   ├── models.py               #   SmashAction / SmashObservation (Pydantic)
│   ├── menu_nav.py             #   Automated menu navigation
│   └── melee_constants.py      #   Stage geometry + velocity limits
│
├── train.py                    # Puff PPO trainer (sim)
├── mango_trainer.py            # Fox/Mango PPO trainer (sim)
├── opponents.py                # Load frozen opponent checkpoints
│
└── checkpoints/                # Trained models
    ├── mango_final.pt          #   Sim-trained Fox
    ├── puff_final.pt           #   Sim-trained Puff
    └── dolphin_fox_final_*.pt  #   Dolphin fine-tuned Fox

Checkpoints

Checkpoint format: {"model_state_dict": ..., "optim_state_dict": ..., "total_frames": int}

Prerequisites