self-driving-rl-v1

Self-Driving RL v1.0

Watch the frozen driver make live decisions in the interactive Hugging Face Space. Each viewer gets an independent hard-mode, dynamic-traffic session running locally in their browser; it is not a prerecorded replay.

This model package contains the two learned layers of the recommended Self-Driving RL driver:

• model.zip: the validation-selected checkpoint from a 2.5M-step V5 QR-DQN run;
• override_model.pt: the calibrated V6 confidence-gated preference residual.
• v5-qrdqn.onnx and v6-rlaif-override.onnx: parity-checked browser exports used by the public live Space.

The complete driver also requires the v1.0 repository code. Its V7/V8 LongitudinalIntentPolicy adds deterministic persistent speed intent, smooth braking, and safety logic. No DAgger weights are included: the evaluated DAgger candidates were rejected by the matched-seed promotion gate and are not part of the recommended driver.

License: the code, documentation, model.zip, and override_model.pt are released under the MIT License; see LICENSE. This is the official model package for slicedonions/self-driving-rl-v1.

Model details

• Simulator observation: 33 normalized float values (nine ego/route values plus six values per lane across four lanes).
• Simulator action: 9 discrete joint actions: left/keep/right × brake/coast/gas.
• Base network: [256, 256] with 64 quantiles.
• Base run: QR-DQN, 2,500,000 steps, training seed 7, eight parallel hard-mode environments, random mirror augmentation, 45-second routes.
• Checkpoint selection: periodic validation selected model.zip; the final live checkpoint was more assertive and less safe, so it was not packaged.
• Preference layer: 35 road/context values plus the one-hot V5 proposal. The 35 values are the 33-value observation and two lane-change memory features.
• V6 gates: calm threshold 0.80, passing threshold 0.50; calibration used development seed 80000 and held-out seed 130000.

The full V5 configuration is in config/v5-training-config.json and V6 provenance is in config/v6-override-provenance.json.

Intended uses

• teaching reinforcement-learning evaluation, reward design, checkpoint selection, and distribution shift;
• reproducing the frozen policy in the bundled Neon Highway simulator;
• studying layered learned/deterministic control and confidence-gated preference residuals;
• comparing new policies on fixed, held-out simulator seeds.

Unsupported uses

• any real vehicle, robot, actuator, driver-assistance system, or safety-critical controller;
• claims about autonomous-driving safety, regulatory compliance, or road readiness;
• inputs from cameras, lidar, maps, GPS, real traffic, weather, pedestrians, or another simulator without retraining and independent validation;
• loading the artifacts from an untrusted or checksum-mismatched source.

Install

Install the public package with:

git clone https://github.com/becauseno7/Self-Driving-RL.git
cd Self-Driving-RL
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e . huggingface_hub
hf download slicedonions/self-driving-rl-v1 model.zip override_model.pt --local-dir models

Verify both downloads before use:

model.zip
  bytes: 3594202
  sha256: 5780BBEE5CE2009459F3AA796AA4982FBF33222DCC182883D31AFAA16C597039

override_model.pt
  bytes: 35439
  sha256: 06C3A0CEE04AAF6B8822781FE78F867F513A3019EBBF7FD8D91E10F117146BEC

Loading and watching

The supported high-level loading example is the project CLI:

sdr-rlaif watch \
  --base-model models/model.zip \
  --override-model models/override_model.pt \
  --longitudinal-intent --dynamic-traffic \
  --difficulty hard --device cpu --episodes 10

The equivalent Python composition is:

from pathlib import Path

from self_driving_rl.longitudinal import LongitudinalIntentPolicy
from self_driving_rl.rlaif import load_override_policy

preference_policy = load_override_policy(
    Path("models/model.zip"),
    Path("models/override_model.pt"),
    device="cpu",
)
driver = LongitudinalIntentPolicy(preference_policy)
# action = int(driver(observation))  # observation.shape == (33,)

Stable-Baselines3 model ZIPs can deserialize cloudpickle. That is a trusted code-loading boundary: use only approved artifacts, verify the SHA-256 digest, and prefer a minimally privileged environment. The override loader uses torch.load(..., weights_only=True).

Evaluation

The final release evaluation ran the complete stack on 100 contiguous, previously unused seeds (690000–690099) in hard mode with dynamic traffic. Every route was 45 simulated seconds.

The raw result is in evaluation/v8-braking-final-unseen-100.json. Reproduce it with:

sdr-rlaif override-evaluate \
  --base-model model.zip \
  --override-model override_model.pt \
  --longitudinal-intent --dynamic-traffic \
  --difficulty hard --device cpu \
  --episodes 100 --seed 690000

Training and preference provenance

V5 was trained in the custom NeonHighwayEnv-v5 task with QR-DQN from sb3-contrib==2.9.0, PyTorch 2.13.0+cu126, Gymnasium 1.3.0, a replay size of 750,000, batch size 512, gamma 0.995, learning rate 0.00015, Polyak factor 0.02, four gradient steps per environment step, and exploration from 1.0 to 0.02 over 30% of training. Periodic 100-route validation selected the packaged model.

V6 kept that checkpoint frozen. Its preference source comprised 96 matched trajectories, 160 comparisons, and 128 individually reviewed AI labels using an ordered safety/completion/passing/calmness rubric. A residual learned leave, calm-correction, or pass-correction choices from 54,000 completed-trajectory states. Confidence calibration and an independent front/rear gap and TTC shield limit deployment-time interventions.

Limitations and provenance caveats

• The 100-route result is a deterministic simulator evaluation, not a confidence bound for real driving or every simulator seed.
• The policy sees privileged structured kinematics rather than raw perception.
• Traffic behavior and challenge generation are authored and much narrower than real traffic.
• The selected checkpoint, thresholds, reward terms, and controller rules were informed by earlier development ranges. The 690000-690099 range was reserved for the reported final run.
• RLAIF labels are small-scale, rubric-dependent, and AI-generated/reviewed; they do not represent a broad human preference population.
• V7/V8 behavior lives in repository code and therefore must be paired with the v1.0 source release; the two weight files alone do not reproduce the result.
• DAgger experiments are documented, but every candidate remained HOLD; none is silently bundled here.
• Exact reproducibility still depends on the pinned Python stack, platform numerics, and repository source. Use the matching v1.0.0 source tag.

See artifact-manifest.json for the machine-readable file inventory and the project README for the architecture and learning guides.