LivePhoto-Wan2.1

LivePhoto-Wan2.1

Code Repository | Project Page | Wan2.1 Model | Paper

The model supports text-guided image-to-video generation with control over motion intensity levels. Built upon the Wan2.1-T2V-1.3B architecture, it is adapted for image-to-video tasks using Pusa fine-tuning strategy. A motion intensity module is plugged in to adjust the movement strength in the generated videos.

<table align="center"> <tr> <td> <img src="./assets/teaser2.gif"> </td> </tr> <tr> </table>

Installation

conda create -n livephoto python=3.10 -y
conda activate livephoto
cd ./LivePhoto-Wan2.1/PusaV1
pip install -e .
pip install xfuser>=0.4.3 absl-py peft lightning pandas deepspeed wandb av 

Model Preparation

pip install -U "huggingface_hub[cli]==0.34.0"
hf download Wan-AI/Wan2.1-T2V-1.3B Wan2.1_VAE.pth models_t5_umt5-xxl-enc-bf16.pth --local-dir ./model_zoo/Wan2.1/base/
hf download Wan-AI/Wan2.1-T2V-1.3B --include="google/*" --local-dir ./model_zoo/Wan2.1/base
hf download shirley430316/LivePhoto-Wan2.1 lora.safetensors base.safetensors --local-dir ./model_zoo/Wan2.1/

After proper preparation, the directory looks like:

./model_zoo
  - Wan2.1
    - base
      - Wan2.1_VAE.pth
      - models_t5_umt5-xxl-enc-bf16.pth
      - google
    - base.safetensors
    - lora.safetensors

Usage Example

I2V with Motion Intensity Levels

CUDA_VISIBLE_DEVICES=0 python examples/pusavideo/wan_14b_multi_frames_pusa.py \
  --image_paths "./demos/input_image.jpg" \
  --prompt "A cute orange kitten with big round eyes stands upright on its hind legs on a smooth wooden floor. The kitten begins to move its tiny front paws up and down rhythmically, swaying its body left and right as if dancing. Its fluffy tail flicks slightly behind it, and the soft lighting creates a warm, cozy indoor atmosphere. The kitten’s ears twitch gently as it keeps its balance, adding to the charm of its playful little dance. The background stays softly blurred, keeping focus on the kitten’s adorable movements." \
  --cond_position "0" \
  --noise_multipliers "0" \
  --lora_path "./model_zoo/Wan2.1/lora.safetensors" \
  --lora_alpha 1.2 \
  --num_inference_steps 30 \
  --cfg_scale 5 \
  --motion_intensity 6  # valid motion intensity levels from 1 through 6

Limitations

• Video generation quality is inherently limited by the capabilities of the base Wan2.1-T2V-1.3B model, e.g., camera motion control is not currently supported.
• Certain image types are likely to generate low quality videos, e.g. cartoon and animated contents, possibly due to dataset biases.

Acknowledgement

This version is developed upon the codebase of Pusa-VidGen. Much appreciation for the insightful project.