castle-nav-geodesic-dqfd

CastleNav-DDQN

CastleNav is a compact deep-reinforcement-learning benchmark modeled as four castle rooms connected by narrow doors. The agent must find a key and then reach a portal. Starts, key positions, and portal positions vary between episodes.

Method

• dueling Q-network;
• Double-DQN targets;
• replay memory;
• target-network updates;
• epsilon-greedy exploration;
• potential-based distance shaping;
• held-out deterministic evaluation.

uv run python projects/castle-nav-rl/train.py
uv run python projects/castle-nav-rl/evaluate.py

The trained policy is stored as Safetensors alongside a JSON architecture config, training metrics, deterministic evaluation results, and example rollouts.

Ablations and final result

Four policy variants were trained:

1. plain Dueling Double-DQN;
2. shortest-path reward shaping;
3. demonstration pretraining with continual imitation loss;
4. geodesic action features plus demonstrations and online RL.

The final 35,909-parameter policy achieved 100% success over 500 held-out episodes with an average path length of 20.886 steps. Earlier variants are preserved as ablations rather than presented as successful models.

MicroPolicy distillation

The final teacher generated 55,286 observations through a mixture of greedy and random exploration. A two-hidden-layer student learned its soft Q distribution and greedy actions.

The student is 20.36 times smaller and retained the teacher's full deterministic success rate on a separate 500-episode seed stream.

uv run python projects/castle-nav-rl/distill_policy.py