reinforce-Pixelcopter-PLE-v0

REINFORCE playing Pixelcopter

A genuinely trained course project for Kay Zheng (bestdive), with AI coding and execution assistance. All training used free local compute. No third-party pretrained weights were used.

Course-format evaluation

The model-index reports 10 stochastic categorical-action episodes, matching the course notebook's example evaluation count and action selection. The seed was fixed to 20260912 before checkpoint selection completed. All episodes, without filtering, appear in course-evaluation.json.

Mean original reward 35.4, population standard deviation 26.2, mean minus standard deviation 9.2. Episodes use the original PLE reward and a 10,000-step maximum. This result is a small-sample course evaluation, not a guarantee of robust performance.

Important larger-sample result

A separate 200-episode deterministic/greedy-action evaluation, seeds 6000000 through 6000199, scored 42.765 +/- 38.82048138547486. Its mean minus standard deviation is 3.944518614525137, below the course threshold of 5. The complete audit is published as independent-evaluation.json. The stochastic and greedy action-selection policies are different; the larger audit still demonstrates that stability needs improvement. No test episodes or unfavorable results were removed.

Training and checkpoint selection

The base model was trained from scratch with REINFORCE and a learned Monte Carlo baseline for 5,182,802 steps / 25,600 episodes, seed 42. Observation engineering uses only observable position, velocity, terrain clearance, and obstacle coordinates. The network is a 10-input, two-layer 64-unit tanh policy with two action logits.

A second, on-policy REINFORCE phase warm-started this own model and optimized a training-only survival objective: reaching 30 points of original reward. The selected checkpoint adds 763,736 training steps, at batch 80 (81 batches of 64 episodes). Total steps on the selected model lineage: 5,946,538. Evaluation does not use this training objective; it always uses original full-episode reward.

Selection used only the mean-minus-standard-deviation score on 100 validation seeds 50000-50099. Training continued to batch 160 without validation improvement, so the batch-80 checkpoint was retained. The logs include those later unused training steps. Test results were not used to choose between checkpoints. Protocol, metadata, training sources and full logs are included.

Reproduction

Use Python 3.12, torch 2.14.0, numpy 1.26.4, gym 0.26.2, pygame 2.6.1, PLE and gym-games. Load policy.pt into the network in evaluate_model.py; the feature transform and default stochastic action selection are included there. PLE's legacy field next_gate_block_top actually describes the block center; preprocessing accounts for this.

See the official Unit 4 notebook. This is a learning project; stronger generalization remains future work.