bevfusion

BEVFusion for Autoware (bevfusion)

3D object detection models for LiDAR-only and camera-LiDAR fusion, used by the autoware_bevfusion node in Autoware.

The models follow the BEVFusion [1] architecture (MIT Han Lab) and run with TensorRT inside Autoware. They are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from the ONNX files on first launch. The sparse convolution backend corresponds to spconv, executed at inference time through the spconv_cpp TensorRT plugins that Autoware installs automatically in its setup script.

Model overview

Variants in this repository

Common model parameters for both variants (from the ml_package_*.param.yaml files): point cloud range [-122.4, -122.4, -3.0, 122.4, 122.4, 5.0] m (roughly 120 m detection radius, matching the t4base_120m family name), voxel size [0.17, 0.17, 0.2] m, max_points_per_voxel: 10, num_proposals: 500, out_size_factor: 8, use_intensity: false.

Pre-processing (voxelization, multi-frame densification, optional image undistortion) and post-processing (circle NMS, IoU NMS, yaw normalization, distance-based score thresholding, area-based class remapping) run in the node, not in the ONNX graphs.

Files

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Inputs

Output is ~/output/objects (autoware_perception_msgs/msg/DetectedObjects): oriented 3D boxes with class and score. The node also publishes per-stage processing-time debug topics.

Usage in Autoware

The node expects these artifacts in $HOME/autoware_data/ml_models/bevfusion/ and launches with, e.g.:

ros2 launch autoware_bevfusion bevfusion.launch.xml \
  model_name:=bevfusion_lidar \
  model_path:=$HOME/autoware_data/ml_models/bevfusion

model_name selects the variant (bevfusion_lidar, the default, or bevfusion_camera_lidar). Add build_only:=true to build the TensorRT engines from the ONNX files as a one-off pre-task. See the package README for the full parameter reference.

Training

The models were trained on TIER IV's internal database; the training data is not publicly available. The consuming package README documents training on roughly 35k LiDAR frames for 30 epochs. Version-specific training details for this t4base_120m/v2 release are not publicly documented.

Related training and inference resources:

• Sparse convolution: https://github.com/traveller59/spconv
• Sparse convolution TensorRT plugins used by Autoware: https://github.com/autowarefoundation/spconv_cpp

Limitations

• Trained on TIER IV's internal sensor configurations; accuracy on a different LiDAR or camera setup (mounting positions, beam count, camera count and resolution) can drop without fine-tuning.
• Only the five classes above are detected. Other road users fall outside the label set.
• The input point cloud must follow the PointXYZIRC layout defined in autoware_point_types.
• The consuming package notes that full integration of the camera-LiDAR fusion mode into the Autoware pipeline is still future work; the model can be employed without changes as a LiDAR-only detector.

Provenance

Citation

@inproceedings{liu2023bevfusion,
  title     = {BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation},
  author    = {Liu, Zhijian and Tang, Haotian and Amini, Alexander and Yang, Xinyu and Mao, Huizi and Rus, Daniela and Han, Song},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  year      = {2023}
}

References

• [1] Liu et al., "BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation", ICRA 2023, arXiv:2205.13542.
• [2] spconv, sparse convolution library: https://github.com/traveller59/spconv
• [3] spconv_cpp, Autoware's sparse convolution TensorRT plugin implementation: https://github.com/autowarefoundation/spconv_cpp