4d-snake-exp05b-ppo-4x4-from-bc

4d-snake-exp05b-ppo-4x4-from-bc

An MLP policy with two hidden layers of 512 units for 4-dimensional snake on the 4^4 board (256 cells, 8 moves): MaskablePPO fine-tuned from the behaviour-cloned network, no curriculum, 20,000,000 environment steps. From a length-1 start it completes the board in every deterministic evaluation episode, evaluated with the protocol of docs/evaluation.md (100 episodes x 3 seeds, masked evaluate_policy).

Results (eval/summary.json)

How to use

The observation is this repository's 4*C + 2 float vector and the action space its 2*ndim masked moves (docs/game_rules.md), so the checkpoint runs inside snake4d's environment:

git clone https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent.git && cd 4d-snake-reinforcement-learning-agent && uv sync
hf download BurnyCoder/4d-snake-exp05b-ppo-4x4-from-bc best_model.zip --local-dir weights
uv run snake4d evaluate --set model_path=weights/best_model.zip --set size=4 --set ndim=4

# https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html
from sb3_contrib import MaskablePPO
from sb3_contrib.common.maskable.utils import get_action_masks
from snake4d.config import Config
from snake4d.vec_env import make_env

cfg = Config(size=4, ndim=4)
model = MaskablePPO.load("weights/best_model.zip", device="cpu")
env = make_env(cfg, 1, 0)  # one board; observation shape (1, 4*C + 2)
obs = env.reset()
masks = get_action_masks(env)  # the legal moves, one row per board
action, _ = model.predict(obs, action_masks=masks, deterministic=True)

Use deterministic mode: the cloned policy follows a fixed Hamiltonian cycle and sampled off-cycle moves eventually trap the snake (see the results table; analysis: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp05_bc_4x4.md).

Training

• Phase train; experiment file experiments/exp05b_ppo_4x4_from_bc.env; write-up: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent/blob/main/reports/experiments/exp05_bc_4x4.md.
• Resolved configuration (config.json):

{
  "size": 4,
  "ndim": 4,
  "idle_mult": 4,
  "r_food": 1.0,
  "r_death": -1.0,
  "r_win": 10.0,
  "r_step": -0.001,
  "shaping_coef": 0.0,
  "n_envs": 4096,
  "total_timesteps": 20000000,
  "n_steps": 64,
  "batch_size": 8192,
  "n_epochs": 4,
  "gamma": 0.99,
  "gae_lambda": 0.95,
  "lr_start": 0.0001,
  "lr_end": 1e-05,
  "clip_start": 0.1,
  "clip_end": 0.05,
  "ent_coef": 0.0,
  "vf_coef": 0.5,
  "max_grad_norm": 0.5,
  "target_kl": 0.03,
  "net_width": 512,
  "device": "auto",
  "torch_threads": 8,
  "seed": 0,
  "curriculum": 0,
  "curriculum_window": 4,
  "curriculum_delta": 0,
  "curriculum_rho": 0.9,
  "curriculum_min_eps": 500,
  "p_true_start": 0.2,
  "eval_episodes": 100,
  "eval_every": 2097152,
  "ckpt_every": 8388608,
  "eval_seeds": "0,1,2",
  "bench_steps": 200000,
  "bc_epochs": 20,
  "bc_lr": 0.001,
  "runs_dir": "runs",
  "run_name": "exp05b_ppo_4x4_from_bc",
  "model_path": "runs/20260904-221730_imitate_exp05_bc_4x4/bc_model.zip",
  "policy": "route"
}

Provenance

• Code: https://github.com/BurnyCoder/4d-snake-reinforcement-learning-agent at commit 016b5dc97583b7c086ad172226997e6aeea92dca.
• Library versions (versions.json): torch 2.14.0+cu130, gymnasium 1.3.0, stable-baselines3 2.9.0, sb3-contrib 2.9.0, numpy 2.5.2, pygame-ce 2.5.8, cuda_device NVIDIA GeForce RTX 5070 Laptop GPU.
• eval/summary.json and eval/eval_episodes.csv are the files the repository's reports quote; every evaluated network is compared in reports/networks.md.
• Collection: https://huggingface.co/collections/BurnyCoder/4d-snake-rl-all-evaluated-networks-6a9d0a0a66c7efcd101b7741

Files

• best_model.zip: the evaluated checkpoint in Stable-Baselines3's save format (policy weights and optimizer state, https://stable-baselines3.readthedocs.io/en/master/guide/save_format.html).
• config.json, versions.json: the run's resolved configuration and environment.
• eval/: evaluation summary and one row per evaluation episode.
• train/progress.csv: the SB3 training log; figures/: the learning curves and the fill histogram.

Licence

MIT, like the repository.