rover-line-follower-ppo

Line-Following PPO Policy (LineFollowerReal-v0)

PPO policy trained in MuJoCo to drive a differential-drive rover ("Mini Claw STEM Rover") along a black line track, for eventual deployment on the real rover via a laptop bridge. Trained in rover_mujoco.

Observation space (Dict)

• image: 64x64 grayscale onboard camera frame, refreshed at 10 Hz (matches the real rover's command rate), under domain randomization (white balance, brightness, pixel noise, effective resolution).
• encoders: left/right wheel encoder ticks accrued since the last control step (quantized, ENCODER_CPR_WHEEL=680, matching the real firmware's raw quadrature-count query) — not velocity; the real rover has no velocity sensor.
• No ground-truth position anywhere in the observation.

Action space

Box(-10, 10, shape=(2,)): left/right wheel angular velocity target, rad/s. Known, unresolved gaps before real deployment:

• The real firmware's "m" UDP command takes left_mps/right_mps (m/s), not rad/s — needs a wheel-radius conversion in the deployment bridge.
• Sim's "same sign on both wheels" spins in place; the real rover's convention is same sign = forward (confirmed from wmala2/rover-firmware). The deployment bridge must negate one channel.

Reward

PROGRESS_WEIGHT * (forward arc-length delta / track length) + CENTER_WEIGHT * (1 - |line-centering error|) + COMPLETION_BONUS on lap/traversal completion. Progress is ground-truth position projected onto the track's waypoints (reward-only privilege — never in the observation), windowed around the previous match to avoid teleporting to a spatially-close-but-path-distant part of the track. See envs/tasks/line_follower_env.py for the full reasoning and the specific failure modes (a "vibrate in place" exploit, and a reward-spike bug) this design fixes.

Training setup

• Algorithm: PPO (MultiInputPolicy, stable-baselines3). SB3 defaults for everything except ent_coef=0.01 (SB3 default is 0.0) — a hyperparameter-sweep result: an initial run with default hyperparameters (scripts/train_real.py, 10M steps) plateaued around ep_rew_mean~90-105 with zero laps/finishes and perfectly reproducible (deterministic) rollouts, indicating premature convergence to an overly-cautious policy. A small sweep (scripts/sweep_hparams.py) found ent_coef=0.01 broke past that plateau immediately (3M steps already exceeded the baseline's 10M-step peak reward), so training continued from that checkpoint (scripts/continue_sweep_winner.py) to 10M total steps.
• n_steps=2048, batch_size=64, n_epochs=10, learning_rate=3e-4 (constant), clip_range=0.2.
• 8 parallel envs (SubprocVecEnv, fork start method), GPU (device="auto").
• Total timesteps: 10,000,000 (summed across all parallel envs; 3M initial + 7M continued).
• Control-loop rate: 10 Hz (CONTROL_HZ), decimated from a 500 Hz physics integrator.
• Domain randomization: action noise/latency, wheel-floor friction, encoder noise, camera FOV/brightness/white-balance/pixel-noise/effective-resolution — see docs/rl-line-follower.md's DR table for exact ranges and reasoning.

Training run summary

Final policy after fixing multiple env/reward bugs (control decimation removing a vibrate-in-place exploit, windowed track-progress reward removing spurious reward spikes, tick-based encoder observation matching real hardware) and a hyperparameter sweep. 10M total timesteps (SB3 PPO default hparams except ent_coef=0.01, found via sweep after a baseline run with ent_coef=0.0 plateaued at ep_rew_mean~90-105 with zero laps). Final eval (10 trials/track, domain_randomize on and off): oval track reliably completes 2 full laps per 300-step episode (reward ~400-430, 100% of trials); s-curve reaches 49-68% of the track under domain randomization (not yet a full finish). Zero line-loss truncations across all eval trials. Never validated on real hardware -- see README for known action-format/sign-convention gaps in the deployment bridge.

Files in this repo

• model.zip — the trained SB3 policy (PPO.load("model.zip"))
• source/ — snapshot of the env/reward/motor-model/training-script source at the time this policy was trained, for exact reproducibility
• This README

Caveats

Never validated on real hardware yet. Domain randomization narrows the sim-to-real gap; it doesn't close it. See docs/rl-line-follower.md in the source snapshot for the full list of open items (action format/sign convention, camera hardware location still unconfirmed, motor model is datasheet-derived not bench-measured).