StreamPETR_nuCarla

StreamPETR — nuCarla

Camera-only 3D object detection for autonomous driving. Six surround-view cameras in, 3D bounding boxes out, with temporal memory carried across frames.

Recommended: full-town epoch-50 checkpoint

The latest checkpoint was trained on the complete official nuCarla split (700 training scenes / 28,000 frames across all towns) and selected by six-class nuCarla NDS on the 150-scene / 6,000-frame validation split.

hf download shin0412/StreamPETR_nuCarla \
  checkpoints/best_full_towns_epoch50_NuCarla_NDS_0.7633_iter_116650.pth

The full-town recipe, resolved config, metrics and matching deployment manifest are under configs/ and full_towns_epoch50/. The model preserves the official ten-class StreamPETR topology; the reported nuCarla score evaluates the six classes present in nuCarla.

Original Town04 release

The original Town04-only checkpoint and its ONNX/TensorRT deployment package remain available below for reproducibility. Those top-level deployment/ artifacts belong to the Town04 checkpoint, not the recommended full-town one.

Trained on the nuCarla Town04 subset. Ships as a PyTorch checkpoint plus ready-to-run ONNX graphs and TensorRT FP16 engines.

Demo

demo/carla_live_detection.mp4 — the model run over a clip recorded live in CARLA 0.9.16: an ego vehicle driving on autopilot through 150 spawned vehicles and 16 pedestrians in Town04. Six camera views with projected 3D boxes plus a bird's-eye panel. Averages 6.7 detections per frame.

The clip is unlabelled, so it is a qualitative check, not a measured score.

Checkpoints

Intermediate epoch checkpoints are not published — they are rotation artifacts from max_keep_ckpts, not separately evaluated models.

Files

checkpoints/                               the three above
stream_petr_r50_nucarla_town04.py          training config
DEPLOYMENT.md                              full inference guide — read this
demo/carla_live_detection.mp4         12M  detection demo
deployment/
  stream_petr_6cam_encoder.onnx        95M
  stream_petr_6cam_temporal_head.onnx  47M
  *_fp16.engine                             TensorRT 10.9, RTX 5070 Ti
  streampetr_trt_runner.py                  runtime, handles temporal memory
  carla_stream_petr_postprocess.py          decoder
  build_streampetr_engines.py               rebuild engines for another GPU
  test_inputs/, reference_outputs/          one frame, to verify your setup

Quick start (TensorRT)

pip install tensorrt==10.9.0.34 numpy torch

import numpy as np
from streampetr_trt_runner import StreamPETRRunner
from carla_stream_petr_postprocess import decode_stream_petr

runner = StreamPETRRunner(
    "deployment/stream_petr_6cam_encoder_fp16.engine",
    "deployment/stream_petr_6cam_temporal_head_fp16.engine",
)

runner.reset()                       # call at the start of every sequence
for frame in sequence:
    out = runner(
        images=frame.images,         # [1, 6, 3, 256, 704] float32, normalized
        timestamp=frame.seconds,
        ego_pose=frame.lidar2global,       # [1, 4, 4]
        ego_pose_inv=frame.global2lidar,   # [1, 4, 4]
    )
    det = decode_stream_petr(
        out["class_logits"], out["bbox_predictions"], score_threshold=0.35
    )

The engines are tied to the GPU and TensorRT version they were built on. Rebuild from the ONNX for anything else — it takes about 15 seconds:

python deployment/build_streampetr_engines.py --deployment-dir deployment --cameras 6

Camera rig — must match

The 3D position embedding is baked into the exported graph. Images from a different camera layout produce meaningless geometry with no error raised. Offsets are ego-frame metres (x forward, y left, z up), yaw counter-clockwise:

All six share 1600×900 at 65° horizontal FOV (fx = fy = 1255.7484, cx = 800, cy = 450) — what CARLA emits for image_size_x=1600, image_size_y=900, fov=65.

Preprocessing, tensor shapes and the temporal memory protocol are all in DEPLOYMENT.md.

Results

nuCarla Town04 validation, 29 scenes / 1160 frames:

Pedestrian scores near zero under 3D IoU because the metric is brutal on small boxes: a 0.42 m wide pedestrian tolerates ~0.25 m of lateral error to reach IoU 0.25, against ~1.2 m for a car. Under the nuScenes center-distance metric the paper actually reports, pedestrian AP is 0.309 and overall mAP 0.4638 — the model does detect pedestrians.

Not comparable to the nuCarla paper's 0.745

The paper reports mAP 0.745 for PETR, but under different conditions:

Town04 is also CARLA's highway map, where objects sit far from the ego (cars average 50 m away), making it harder than the urban maps in the full set.

Training

• Initialized from the official nuScenes StreamPETR checkpoint. All six target classes exist in nuScenes' ten, so the classification rows were sliced across rather than randomly re-initialized — nothing in the network starts from scratch. This lifted first-epoch mAP from 0.0497 to 0.2701.
• 50 epochs planned, batch 12, AdamW at 1.2e-4 with cosine annealing, FP16. The run reached epoch 45; mAP had plateaued since epoch 23 and the LR was annealed to 3.6e-6.
• Best checkpoint at iteration 8512 (epoch 28).

Verification

The ONNX wrapper reproduces the native PyTorch head exactly (max |Δ| = 0.0 on both outputs). TensorRT FP16 against PyTorch FP32 on the bundled frame:

Decoded detections agree: 17 vs 17 above score 0.2, every one placed within 0.02–0.10 m. Two pairs swap rank order, both between detections whose scores tie to within 0.003.

Known FP16 effect. 768 of the 1024 memory slots are bit-exact frame to frame; within the 256 slots selected fresh each frame, FP16 scoring picks a slightly different subset near the top-256 cut-off. Its accumulation over long sequences has not been measured — build the FP32 engine if you need to rule it out.

Limitations

• Trained on one map (Town04). Expect degradation on urban maps or dense intersections.
• Pedestrian localization is weak in absolute terms; suitable for perception research, not safety-critical use.
• Simulation only — no real-world images were used, and no sim-to-real transfer has been evaluated.
• The rig is fixed. Any change to camera placement, count or intrinsics requires re-exporting from the checkpoint.

Citation

@article{nucarla,
  title={nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation},
  author={Qiao, Zhijie and Cao, Zhong and Liu, Henry X.},
  year={2025},
  url={https://arxiv.org/abs/2511.13744}
}

@article{streampetr,
  title={Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection},
  author={Wang, Shihao and Liu, Yingfei and Wang, Tiancai and Li, Ying and Zhang, Xiangyu},
  journal={arXiv preprint arXiv:2303.11926},
  year={2023}
}

License

Apache 2.0, following StreamPETR and the nuCarla dataset.