camera_streampetr

StreamPETR for Autoware (camera_streampetr)

Camera-only multi-view 3D object detection models, used by the autoware_camera_streampetr node in Autoware.

The models follow the StreamPETR [1] architecture, an object-centric temporal modeling approach for efficient multi-view 3D detection, and run with TensorRT inside Autoware. This is the first camera-only 3D object detection node in Autoware: it detects 3D objects from surround-view camera images without any LiDAR input. The models are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from the ONNX files on first launch.

Model overview

The network is split into three ONNX sub-models, matching how the node consumes them:

1. Image feature extractor (backbone): simplify_extract_img_feat.onnx
2. Position embedding: simplify_position_embedding.onnx
3. Detection head with temporal memory: simplify_pts_head_memory.onnx

Pre-processing (undistortion if needed, resize, crop, normalization, ego-vehicle masking) and post-processing (circle NMS, IoU NMS, per-class confidence thresholding, yaw normalization) run in the node, not in the ONNX graphs.

Files

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Inputs

• ~/input/camera*/image (sensor_msgs/msg/Image or sensor_msgs/msg/CompressedImage): camera images, compressed or uncompressed.
• ~/input/camera*/camera_info (sensor_msgs/msg/CameraInfo): camera intrinsics.

The node is optimized for multi-camera systems where camera topics arrive sequentially rather than all at once: each image is preprocessed and stored on the GPU as it arrives, and the forward pass is triggered when the anchor camera's image arrives and all images are synchronized within max_camera_time_diff.

Output

• ~/output/objects (autoware_perception_msgs/msg/DetectedObjects): oriented 3D boxes with class and score.
• latency/* (autoware_internal_debug_msgs/msg/Float64Stamped): per-stage timing topics for debugging. On an RTX 3090 the package reports a total processing time of about 26 ms per inference (the reported time between consecutive predictions is about 111 ms).

Usage in Autoware

The node expects these artifacts in ~/autoware_data/ml_models/camera_streampetr/ (the default model_path) and launches with:

ros2 launch autoware_camera_streampetr streampetr.launch.xml \
  model_path:=$HOME/autoware_data/ml_models/camera_streampetr

Add build_only:=true to build the TensorRT engines from the ONNX files as a one-off pre-task. The ml_package_camera_streampetr.param.yaml shipped here is loaded by the launch file and carries the model-dependent parameters; node behavior parameters live in the package's own config/camera_streampetr.param.yaml. See the package README for the full parameter reference.

Training

The model was trained and exported with the AWML StreamPETR project:

• https://github.com/tier4/AWML/tree/main/projects/StreamPETR

Training datasets, schedules, and evaluation metrics for these specific weights are not publicly documented.

Limitations

• The model only predicts objects inside the detection range above; the node filters out anything beyond it.
• Only the five classes above are detected. Other road users fall outside the label set.
• The node assumes undistorted input images (or enables undistortion itself), accurate camera calibration, cameras synchronized within max_camera_time_diff, and tf transforms between the camera frames, base_link, and map (for ego motion compensation).
• Predictions do not include object velocity.

Provenance

The v1.0 tag corresponds to the streampetr/v1 artifact version at the original hosting path.

Citation

@article{wang2023streampetr,
  title   = {Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection},
  author  = {Wang, Shihao and Liu, Yingfei and Wang, Tiancai and Li, Ying and Zhang, Xiangyu},
  journal = {arXiv preprint arXiv:2303.11926},
  year    = {2023}
}

References

• [1] Wang et al., "Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection", arXiv:2303.11926, 2023.
• AWML StreamPETR project: https://github.com/tier4/AWML/tree/main/projects/StreamPETR
• Autoware: https://github.com/autowarefoundation/autoware