UCM

<!-- ## ___***UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models***___ <div align="center"> _**[Tian-Xing Xu^*1](https://scholar.google.com/citations?user=zHp0rMIAAAAJ), [Zi-Xuan Wang^*1](https://scholar.google.com/citations?user=4JKwZRsAAAAJ), [Guangyuan Wang^*&dagger;2](https://scholar.google.com/citations?user=OYwm0GsAAAAJ&hl=zh-CN), [Li Hu^&dagger;2](https://scholar.google.com/citations?user=Arz3iGUAAAAJ), [Zhongyi Zhang³](https://scholar.google.com/citations?user=dOWmKOUAAAAJ), [Peng Zhang²](https://scholar.google.com/citations?user=QTgxKmkAAAAJ), Bang Zhang^&ddagger;2, [Song-Hai Zhang^&ddagger;1](https://scholar.google.com/citations?user=AWtV-EQAAAAJ)**_ ^\*Co-first authors ^&dagger;Project leaders ^&ddagger;Corresponding authors ¹Tsinghua University ²Alibaba ³UCST  &nbsp;<a href='https://arxiv.org/pdf/2602.22960'><img src='https://img.shields.io/badge/arXiv-2504.01016-b31b1b.svg'></a> &nbsp; <a href='https://humanaigc.github.io/ucm-webpage/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> &nbsp; </div> <div align="center"> <img width="320" alt="pipeline" src="assets/teaser.gif"> <img width="320" alt="pipeline" src="assets/teaser2.gif"> </div> --> <div align="center"> <h2><i><b>UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models</b></i></h2>

Tian-Xing Xu^∗¹^, Zi-Xuan Wang^∗¹^, Guangyuan Wang^∗†²^, Li Hu^†²^, Zhongyi Zhang^³^, Peng Zhang^²^, Bang Zhang^‡²^, Song-Hai Zhang^‡¹^

¹Tsinghua University   ²Alibaba   ³UCST

^∗Co-first authors   ^†Project leaders   ^‡Corresponding authors

     

<br/>

<img width="320" alt="teaser1" src="assets/teaser.gif"/> <img width="320" alt="teaser2" src="assets/teaser2.gif"/>

</div> <!-- ## 🔆 Notice <!-- We recommend that everyone use English to communicate on issues, as this helps developers from around the world discuss, share experiences, and answer questions together. For further implementation details, please contact `[email protected]` or `[email protected]`. For business licensing and other related inquiries, don't hesitate to contact `[email protected]`. --> <!-- If you find UCM useful, **please help ⭐ this repo**, which is important to Open-Source projects. Thanks! -->

📝 Introduction

We present UCM, a novel framework to explore 4D world of a reference image following the user-specified camera trajectory, which unifies long-term memory and precise camera control via a time-aware positional encoding warping mechanism.

Release Notes:

• [2026/08/07] 🔥🔥🔥UCM is released now, have fun!

🚀 Quick Start

Installation

1. Clone this repo:

git clone --recursive https://github.com/HumanAIGC/UCM.git

<!-- TODO -->

1. Install dependencies (please refer to requirements.txt):

pip install -r requirements.txt

Model Download

Download models using huggingface-cli:

pip install "huggingface_hub[cli]"
huggingface-cli download HumanAIGC-Team/UCM --local-dir ./workspace/pretrained/

Download models using modelscope-cli:

pip install modelscope
modelscope download --model DAMOXR/UCM --local_dir ./workspace/pretrained/

Inference

Run inference code on our provided demo videos, which requires a GPU with ~42GB memory and ~5min to generate a 12s video (241 frames):

python main.py \
    --img_path examples/images/frame_0000.png \
    --traj_path examples/cameras/cameras_0000.json \
    --prompt "The video captures a serene and picturesque scene of a traditional Dutch village on a bright, sunny day. The sky is a vibrant blue with scattered white clouds, creating a perfect backdrop for the charming architecture and lush greenery. The camera pans slowly across the village, revealing a row of quaint houses with red-tiled roofs and brick facades, typical of Dutch design. Some houses have green-painted wooden shutters and doors, adding a touch of color to the scene.  A narrow cobblestone street runs through the village, lined with parked cars on both sides, indicating a peaceful residential area."

To obtain all demo videos, you can use the following instruction:

python main.py --metafile examples/examples.csv

Parameters

• --img_path: Path to your reference image.
• --traj_path: Path to your specific camera trajectory file (.json).
• --prompt: Text prompt.
• --save_folder: Path to your folder for saving generated videos.
• --camera_scale_factor: Scales the camera center within the trajectory to match the scale of the 3D scene representation.
• --num_denoising_steps: The number of denoising iterations. 20 reaches a balance for the Gradio demo. 50 is used in our paper.
• --guidance_scale: Classifier-Free Guidance scale. The default value 5.0 is recommended.
• --seed: Seed for initializing the random number generator, controlling the randomness of Gaussian noise sampling.
• --duration: Only the first Duration frames of the camera trajectory will be processed. -1 represents the whole trajectory.

🤖 Gradio Demo

<!-- - Online demo: [**Huggingface Demo**](https://huggingface.co/spaces/) TODO -->

• Local demo:

  gradio app.py
  

🤝 Contributing

• Welcome to open issues and pull requests.
• Welcome to optimize the inference speed and memory usage, e.g., through model quantization, distillation, or other acceleration techniques.

❤️ Acknowledgement

We have used codes from other great research work, including STream3R and Wan2.1. We sincerely thank the authors for their awesome works!

📄 License

This project is licensed under the Apache License 2.0.

📜 Citation

If you find this work helpful, please consider citing:

@article{xu2026ucm,
  title={UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models},
  author={Xu, Tianxing and Wang, Zixuan and Wang, Guangyuan and Hu, Li and Zhang, Zhongyi and Zhang, Peng and Zhang, Bang and Zhang, Songhai},
  journal={arXiv preprint arXiv:2602.22960},
  year={2026}
}

<!-- TODO -->