BEV-VAE

Scalable and Generalizable Autonomous Driving Scene Synthesis

Zeming Chen, Hang Zhao.

Abstract

<details> <summary><b>TL; DR</b> We introduce BEV-VAE, a variational autoencoder that unifies multi-view images into a BEV representation for scalable and generalizable autonomous driving scene synthesis. </summary> Generative modeling has shown remarkable success in vision and language, inspiring research on synthesizing autonomous driving scenes. Existing multi-view synthesis approaches commonly operate in image latent spaces with cross-attention to enforce spatial consistency, but they are tightly bound to camera configurations, which limits dataset scalability and model generalization. We propose BEV-VAE, a variational autoencoder that unifies multi-view images into a compact bird’s-eye-view (BEV) representation, enabling encoding from arbitrary camera layouts and decoding to any desired viewpoint. Through multi-view image reconstruction and novel view synthesis, we show that BEV-VAE effectively fuses multi-view information and accurately models spatial structure. This capability allows it to generalize across camera configurations and facilitates scalable training on diverse datasets. Within the latent space of BEV-VAE, a Diffusion Transformer (DiT) generates BEV representations conditioned on 3D object layouts, enabling multi-view image synthesis with enhanced spatial consistency on nuScenes and achieving the first complete seven-view synthesis on AV2. Finally, synthesized imagery significantly improves the perception performance of BEVFormer, highlighting the utility of scalable and generalizable scene synthesis for autonomous driving. </details>

Method

Overall architecture of BEV-VAE with DiT for multi-view image generation.

In Stage 1, BEV-VAE learns to encode multi-view images into a spatially compact latent space in BEV and reconstruct them, ensuring spatial consistency. In Stage 2, DiT is trained with Classifier-Free Guidance (CFG) in this latent space to generate BEV representations from random noise, which are then decoded into multi-view images.

Experiments

Datasets

This study uses four multi-camera autonomous driving datasets that differ substantially in scale, camera configuration, annotated categories, and recording locations. Despite these differences, all datasets provide full 360° coverage of the surrounding scene.

We introduce a new hybrid autonomous driving dataset configuration, <b>PAS</b>, which combines nu<b>P</b>lan, <b>A</b>V2, and nu<b>S</b>cenes.

Multi-view Image Reconstruction

BEV-VAE learns unified BEV representations by reconstructing multi-view images, integrating semantics from all camera views while modeling 3D spatial structure. Reconstruction metrics provide an indirect evaluation of the quality of the learned BEV representations. For reference, we compare with SD-VAE, a foundational model trained on LAION-5B, which encodes a single $256\times256$ image into a $32 \times32\times4$ latent. In contrast, BEV-VAE encodes multiple $256\times256$ views into a $32\times32\times16$ BEV latent, facing the more challenging task of modeling underlying 3D structure.

Reconstruction metrics on nuScenes compared with SD-VAE.

Reconstruction metrics on AV2 compared with SD-VAE.

SD-VAE focuses on per-view image fidelity, whereas PAS-trained BEV-VAE achieves superior multi-view spatial consistency (MVSC).

Multi-view image reconstruction on nuScenes

<b>Click the image below</b> to watch the ego view rotate 360° horizontally.

Multi-view image reconstruction on AV2

<b>Click the image below</b> to watch the ego view rotate 360° horizontally.

Multi-view image reconstruction on nuPlan

<b>Click the image below</b> to watch the ego view rotate 360° horizontally.

Novel View Synthesis

<b>Novel view synthesis via camera pose modifications on nuScenes.</b> Row 1 shows real images from the nuScenes validation set, and Rows 2-3 show reconstructions with all cameras rotated 30° left and right, where the cement truck and tower crane truck remain consistent across views without deformation.

<b>Novel view synthesis cross camera configurations.</b> Row 1 presents real images from the nuPlan validation set. Row 2 and Row 3 show reconstructions using camera parameters from AV2 and nuScenes, respectively. The model captures dataset-specific vehicle priors: AV2 reconstructions include both the front and rear of the ego vehicle, while nuScenes reconstructions mainly show the rear (with the rightmost image corresponding to the rear-view camera for alignment).

Zero-shot BEV Representation Construction

<b>Zero-shot BEV representation construction on WS101.</b> Row 1 shows real images from the WS101 validation set. Rows 2 and 3 show zero-shot and fine-tuned reconstructions, respectively, with object shapes preserved in the zero-shot results and further sharpened after fine-tuning.

<b>Zero-shot and fine-tuned reconstruction metrics on WS101 compared with SD-VAE.</b>

Autonomous Driving Scene Synthesis

Autonomous driving scene synthesis from AV2 to nuScenes.

BEV-VAE with DiT generates a BEV representation from 3D bounding boxes of AV2, which can then be decoded into multi-view images according to the camera configurations of nuScenes.

Multi-view image generation on AV2 with 3D object layout editing.

<b>Click the image below</b> to watch the ego view rotate 360° horizontally.

Multi-view image generation on nuScenes with 3D object layout editing.

<b>Click the image below</b> to watch the ego view rotate 360° horizontally.

Data Augmentation for Perception

BEV-VAE w/ DiT using the Historical Frame Replacement strategy (randomly replacing real frames with generated ones) improves BEVFormer’s perception by enabling the model to learn invariance of object locations relative to appearance.