4d-snake-exp03b-ppo-3x4-backplay

4d-snake-exp03b-ppo-3x4-backplay

An MLP policy with two hidden layers of 512 units for 4-dimensional snake on the 3^4 board (81 cells, 8 moves): MaskablePPO trained from scratch, Backplay reverse curriculum (gate 0.9, window 4), 30,000,000 environment steps. From a length-1 start it completes the board in 0.0 % of deterministic episodes, evaluated with the protocol of docs/evaluation.md (100 episodes x 3 seeds, masked evaluate_policy).

Negative result. This network never fills the board from the true start (mean final fill 0.549). It is published so the failure is reproducible; the write-up linked below analyses why.

Results (eval/summary.json)

How to use

The observation is this repository's 4*C + 2 float vector and the action space its 2*ndim masked moves (docs/game_rules.md), so the checkpoint runs inside snake4d's environment:

git clone https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent.git && cd 4d-snake-reinforcement-learning-agent && uv sync
hf download BurnyCoder/4d-snake-exp03b-ppo-3x4-backplay best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=3 --set ndim=4

# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env

cfg = Config(size=3, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0)  # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env)  # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)

Training

• Phase train; experiment file experiments/exp03b_ppo_3x4_backplay.env; write-up: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp03_ppo_3x4.md.
• Resolved configuration (config.json):

{
  "size": 3,
  "ndim": 4,
  "idle_mult": 4,
  "r_food": 1.0,
  "r_death": -1.0,
  "r_win": 10.0,
  "r_step": -0.001,
  "shaping_coef": 0.0,
  "n_envs": 2048,
  "total_timesteps": 30000000,
  "n_steps": 64,
  "batch_size": 8192,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "lr_start": 0.0003,
  "lr_end": 1e-05,
  "clip_start": 0.2,
  "clip_end": 0.05,
  "ent_coef": 0.01,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "target_kl": 0.03,
  "net_width": 512,
  "device": "auto",
  "torch_threads": 8,
  "seed": 0,
  "curriculum": 1,
  "curriculum_window": 4,
  "curriculum_delta": 0,
  "curriculum_rho": 0.9,
  "curriculum_min_eps": 500,
  "p_true_start": 0.2,
  "eval_episodes": 100,
  "eval_every": 1310720,
  "ckpt_every": 5242880,
  "eval_seeds": "0,1,2",
  "bench_steps": 200000,
  "runs_dir": "runs",
  "run_name": "exp03b_ppo_3x4_backplay",
  "model_path": "",
  "policy": "route"
}

Provenance

• Code: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent at commit 9dfaa8a6f662d57fc8a02ee67a15efa3cbefab6c.
• Library versions (versions.json): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
• eval/summary.json and eval/eval_episodes.csv are the files the repository's reports quote; every evaluated network is compared in reports/networks.md.
• Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741

Files

• best_model.zip: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
• config.json, versions.json: the run's resolved configuration and environment.
• eval/: evaluation summary and one row per evaluation episode.
• train/progress.csv: the SB3 training log; figures/: the learning curves and the fill histogram.

Licence

MIT, like the repository.