DualCamCtrl

DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation

Hongfei Zhang ^1*, Kanghao Chen ^1,5*, Zixin Zhang ^1,5, Harold H. Chen ^1,5, Yuanhuiyi Lyu ¹, Yuqi Zhang ³, Shuai Yang ¹, Kun Zhou ⁴, Ying-Cong Chen ^1,2,✉

¹ HKUST(GZ) ² HKUST ³ Fudan University ⁴ Shenzhen University ⁵ Knowin

* Equal Contribution. ✉Corresponding author.

🧩 Contents

1. 📰 News

2. ⚙️ TODO

3. 🎯 Overview

4. 🔧 Installation

5. 🔮 Inference

6. 🔥 Training

📰 News

✅ 2025.11 — Released inference pipeline & demo dataset ✔️

✅ 2025.11 — Uploaded official DualCamCtrl checkpoints to HuggingFace 🔑

⚙️ TODO

⬜ Release the training code 🚀

🎯 Overview

Abstract

This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the SemantIc Guided Mutual Alignment (SIGMA) mechanism, which performs RGB–depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation with over 40% reduction on camera motion errors compared with prior methods.

Results

Comparison between our method and other state-of-the-art approaches. Given the same camera pose and input image as generation conditions, our method achieves the best alignment between camera motion and scene dynamics, producing the most visually accurate video. The ’+’ signs marked in the figure serve as anchors for visual comparison.

Quantitative comparisons on I2V setting. ↑ / ↓ denotes higher/lower is better. Best and second best results highlighted.

Quantitative comparisons on T2V setting across REALESTATE10K and DL3DV.

🔧 Installation

Clone repo and create an enviroment with Python 3.11:

git clone https://github.com/soyouthinkyoucantell/DualCamCtrl.git
conda create -n dualcamctrl python=3.11
conda activate dualcamctrl

Install DiffSynth-Studio dependencies from source code:

cd DualCamCtrl
pip install -e .

Then install GenFusion dependencies:

mkdir dependency
cd dependency
git clone https://github.com/rmbrualla/pycolmap.git 
cd pycolmap
pip install -e .
pip install numpy==1.26.4 peft accelerate==1.9.0 decord==0.6.0 deepspeed diffusers omegaconf  

🔮 Inference

Checkpoints

Get the checkpoints from the HuggingFace repo:  DualCamCtrl Checkpoints

Put it the checkpoints dir

cd ../.. # make sure you are at the root dir 

Your project structure should be like

DualCamCtrl/
├── checkpoints/                 # ← Put downloaded .pt here
│   └── dualcamctrl_diffusion_transformer.pt
├── demo_dataset/                # Small demo dataset strcture
├── demo_pic/                    # Demo images for quick inference
├── diffsynth/                   
├── examples/   
├── ....                 
├── requirements.txt             
├── README.md                    
└── setup.py                    

Test with our demo pictures and depth:

cd .. # make sure you are at the root dir 
export PYTHONPATH=.
python -m test_script.test_demo

🔥 Training

Training details coming soon… Stay tuned! 🚀