PanoWorld

<div align="center" style="margin-top: 0px; margin-bottom: 0px;"> <h1> <img src="assets/helicopter.png" width="64" alt="PanoWorld logo" style="vertical-align: middle; margin-right: 12px;"> PanoWorld: Real-World Panoramic Generation </h1> <p align="center"> <a href="https://lihaoy-ux.github.io/panoworld-page/"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a> <a href="http://arxiv.org/abs/2607.09661"><img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv" alt="arXiv"></a> <a href="https://huggingface.co/Insta360-Research/PanoWorld/"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="Hugging Face"></a> <a href="#"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-World360(coming soon)-yellow" alt="Demo"></a> </p> <p align="center"> Haoyuan Li¹ &nbsp; Dizhe Zhang¹✉ &nbsp; Yuemei Zhou¹ &nbsp; Xiangkai Zhang^1,2 &nbsp; Haoran Feng^1,3 &nbsp; Xiaofan Lin¹ &nbsp; Wenjie Jiang¹ &nbsp; Bo Du⁴ &nbsp; Ming-Hsuan Yang⁵ &nbsp; Lu Qi^1,4✉ <br><br> ¹Insta360 Research &nbsp;&nbsp; ²Institute of Automation Chinese Academy of Sciences &nbsp;&nbsp; ³Tsinghua University <br> ⁴Wuhan University &nbsp;&nbsp; ⁵UC Merced </p>

</div>

Quick try: demo assets in assets/demo/ — see Inference below.

Updates

• [2026-07] 🎉 Initial release of code. Training code is coming soon.

Introduction

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.

Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA). Then, a three-stage training pipeline is introduced to progressively optimize each component.

To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.

Environment Setup

Prerequisites

• OS: Linux (tested on Ubuntu 22.04)
• GPU: CUDA-compatible GPU with ≥20GB VRAM
• CUDA: 12.8 or higher
• Python: 3.10
• FFmpeg: For video processing

Step 1: Create Conda Environment

Action Model inference assumes the PanoWorld conda env is already active.

git clone https://github.com/Insta360-Research-Team/PanoWorld.git
cd PanoWorld

bash scripts/setup_panoworld_env.sh
conda activate PanoWorld

Or install manually:

conda create -n PanoWorld python=3.10 -y
conda activate PanoWorld
pip install -e .
export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}"

Dependencies are listed in requirements.txt. After Step 1, continue with Step 2–3 (base model + PanoWorld checkpoints) before running inference_action.sh.

Step 2: Download Base Model (Wan2.2-TI2V-5B)

PanoWorld is built upon the Wan-AI Wan2.2-TI2V-5B video diffusion model.

# Download using provided script
python scripts/download_wan2.2.py

# Or manually download from Hugging Face
# Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B
# Download to: models/Wan-AI/Wan2.2-TI2V-5B/

Step 3: Download Panoworld Models

Download the 480p or 720p checkpoints:

# Option 1: Using our download script
python scripts/download_panoworld_models.py

# Option 2: Manual download from Hugging Face
# Visit: https://huggingface.co/Insta360-Research/PanoWorld

Final model directory structure:

models/
├── Wan-AI/
│   └── Wan2.2-TI2V-5B/
├── lora/
│   ├── 480p/480p_lora.safetensors
│   └── 720p/720p_lora.safetensors
├── action/
│   ├── 480p/480p_action.safetensors
│   └── 720p/720p_action.safetensors
└── casual/
    ├── dmd_chunkwise/model.pt
    └── rolling_dmd_480p_161/model.pt

Step 4: Install Causal-Forcing Package (Optional)

The Causal-Forcing stage requires additional dependencies:

Please refer to Causal-Forcing/README.md for installation instructions.

World360 Dataset

World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information.

Output: Each dataset generates:

• 81-frame & 161-frame panoramic video.
• Camera trajectory csv file
• PNG image sequence

Inference

Bundled demo assets live under assets/demo/. Resolution-specific cases are under 480/ and 720/; each case folder contains a 2:1 equirectangular panorama (input.jpg), text prompt (prompt.txt), and camera trajectory (pose.txt). 720p cases also include a reference video (reference_gen_joint_step2000.mp4).

Pose spacing: For custom pose.txt trajectories, we recommend a horizontal step of ~0.05 between consecutive frames, i.e. sqrt(action_dx² + action_dy²) ≈ 0.05 in the model world frame (-x forward, +y left). The bundled demo poses use this spacing.

Camera-Controlled Video Generation

High-quality panoramic I2V with camera trajectory control.
Unified entry: inference_action.sh

Output: {output}/gen_video.mp4 (single sample) or gen_video_{i}.mp4 (batch).

# --- Demo: image + prompt + synthetic forward motion ---
./inference_action.sh \
  --resolution 480 \
  --image assets/demo/input.jpg \
  --prompt "$(cat assets/demo/prompt.txt)" \
  --motion forward \
  --output ./inference_output/demo_forward

# --- Demo case with recorded pose (480p) ---
CASE=assets/demo/480/case2_waterway_slice000
./inference_action.sh \
  --resolution 480 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# --- Demo case with recorded pose (720p) ---
CASE=assets/demo/720/case1_waterway_slice706
./inference_action.sh \
  --resolution 720 \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_action

# 480p — single sample from test CSV
./inference_action.sh \
  --resolution 480 \
  --prompt_nums 1 \
  --output ./inference_output/demo_480p

# 720p — native 1408×704, upscaled to 1440×720
./inference_action.sh \
  --resolution 720 \
  --prompt_nums 1 \
  --output ./inference_output/demo_720p

# Or via top-level wrapper
RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh
RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh

python Action-Model/infer_action.py --help

Default test CSVs: data_test.csv (480p), data_test_720p.csv (720p).

---

Causal Forcing Stage

Real-time panoramic generation with Causal Forcing on Wan2.2 5B.
Entry: Casual-forcing/inference_causal.sh (requires causal_forcing env).

Output: {output}/causal_video.mp4

cd Casual-forcing

# --- Quick demo (root assets, 480p case1) ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --output ../assets/demo/out_short

# --- Demo case with recorded pose ---
CASE=../assets/demo/480/case2_waterway_slice000
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --output ${CASE}/out_pose

# --- 161-frame long video (Rolling Forcing) ---
CASE=../assets/demo/480/case1_park_slice002
./inference_causal.sh \
  --image ${CASE}/input.jpg \
  --prompt "$(cat ${CASE}/prompt.txt)" \
  --motion ${CASE}/pose.txt \
  --frames 161 \
  --output ${CASE}/out_long

# --- Synthetic motion presets ---
./inference_causal.sh \
  --image ../assets/demo/input.jpg \
  --prompt "$(cat ../assets/demo/prompt.txt)" \
  --motion forward \
  --output ../assets/demo/out_forward

python infer_causal.py --help

---

Memory Preview

Coming soon. Long-range memory-augmented 720p inference (161-frame) will be released in a future update.

---

Training

PanoWorld training is organized in stages:

Stage 1  Video LoRA     →  output/lora_480p | lora_720p
Stage 2  Action Model   →  output/action_480p | action_720p  (requires Stage 1)
Stage 3  Memory         →  coming soon
Stage 4  Causal Forcing →  Casual-forcing/logs/  (480p, see below)

Use RESOLUTION=480|720 to switch presets (Matrix-3D convention). Config: configs/resolution.yaml.

Training CSVs must include panoramic video paths and text prompts. Override defaults with DATA_CSV=/path/to/your.csv.

---

Stage 1 — Video LoRA

Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module).

# 480p
RESOLUTION=480 bash scripts/train/01_video_lora.sh

# 720p (recommended for Action Model 720p)
RESOLUTION=720 bash scripts/train/01_video_lora.sh

Checkpoints are saved under {OUTPUT}/checkpoints/ with latest.json for resume.

Legacy wrappers: scripts/train/01_video_lora_480p.sh, 01_video_lora_720p.sh.

---

Stage 2 — Action Model

Freeze LoRA and train the cam_self_attn camera-control module (trajectory-conditioned I2V). Requires Stage 1 LoRA.

# After Stage 1 completes
RESOLUTION=480 bash scripts/train/02_action.sh
RESOLUTION=720 bash scripts/train/02_action.sh

# Warm-start from a simulation-pretrained action checkpoint (optional)
INIT_ACTION=/path/to/simulation_action.safetensors \
  RESOLUTION=720 bash scripts/train/02_action.sh

Checkpoints: {OUTPUT}/checkpoints/. Use the latest action weights with inference_action.sh for preview inference.

720p project-specific script (hardcoded paths): Action-Model/train_720p.sh.

---

Stage 3 — Memory

Coming soon.

---

Stage 4 — Causal Forcing

See Casual-forcing/README.md for installation and training.

Checkpoint Layout

output/
├── lora_480p/checkpoints/
├── lora_720p/checkpoints/
├── action_480p/checkpoints/
└── action_720p/checkpoints/
Casual-forcing/logs/          # Causal Forcing training outputs

Project Structure

PanoWorld/
├── assets/demo/                # Demo inputs (480/720 cases: input, prompt, pose)
│   ├── 480/                    # 960×480 cases
│   └── 720/                    # 1408×704 cases (+ reference_gen_*.mp4)
├── inference_action.sh         # Action Model unified entry → gen_video.mp4
├── Action-Model/               # Action Model Python scripts (480p / 720p)
│   └── infer_action.py         # unified CLI
├── Casual-forcing/             # Causal Forcing inference & training
│   ├── infer_causal.py         # unified CLI → causal_video.mp4
│   └── inference_causal.sh
├── diffsynth/                  # Shared DiffSynth modules
├── scripts/train/              # Staged training launchers (01–04)
├── models/                     # Wan2.2 base + PanoWorld checkpoints
│   ├── lora/                   # Video LoRA (480p / 720p)
│   ├── action/                 # Action Model (480p / 720p)
│   └── casual/                 # Causal Forcing inference weights
├── inference_preview.sh        # RESOLUTION=480|720 wrapper
├── utils/
│   ├── run_metrics.sh        # Unified 480p/720p quality evaluation
│   ├── resolution_config.py    # Resolution presets helper
│   └── validate/             # step1_validate, step2_tractory, metric modules
└── configs/resolution.yaml     # 480 / 720 presets

Tools

Panoramic video quality evaluation — unified script for 480p / 720p: utils/run_metrics.sh (implementation: utils/validate/).

Metrics: PSNR (windowed), FID / FID_pole / FID_equ, FAED, NIQE, QAlign (Quality / Aesthetic).

RESOLUTION=480 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh
RESOLUTION=720 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh

Results are written to RESULT_DIR (PSNR CSV + eval_results/eval_results_*.txt).

Acknowledgments

We thank the following projects for their inspiring work, our code is partially based on the code from these projects:

• Wan-AI: Base video diffusion model

• UCPE: Camera-controlled text-to-video generation.

• Causal-Forcing: Causal-Forcing distillation for fast diffusion models

Citation

If you find Panoworld useful for your research, please cite:

@misc{li2026panoworldrealworldpanoramicgeneration,
      title={PanoWorld: Real-World Panoramic Generation}, 
      author={Haoyuan Li and Dizhe Zhang and Yuemei Zhou and Xiangkai Zhang and Haoran Feng and Xiaofan Lin and Wenjie Jiang and Bo Du and Ming-Hsuan Yang and Lu Qi},
      year={2026},
      eprint={2607.09661},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.09661}, 
}