TemporalNet2-stable-diffusion-xl-base-1.0

TemporalNet2 ControlNet for SDXL

This is a TemporalNet2 ControlNet model trained on SDXL (Stable Diffusion XL base 1.0).

Model Description

TemporalNet2 is a ControlNet variant designed for temporal coherence in video generation. It takes two conditioning inputs:

• Previous Frame: The previous frame in the video sequence (3 channels)
• Optical Flow: The optical flow between the previous and current frame (3 channels)

Total conditioning channels: 6 channels

This model was trained to generate temporally coherent frames by learning from both the visual content of the previous frame and the motion information encoded in optical flow.

Usage

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel, EulerDiscreteScheduler
from PIL import Image
import torch

# Load the ControlNet model
controlnet = ControlNetModel.from_pretrained(
    "YOUR_USERNAME/temporalnet2-sdxl-controlnet",
    torch_dtype=torch.float16
)

# Create the pipeline
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    torch_dtype=torch.float16
)
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")

# Load your conditioning images
prev_frame = Image.open("previous_frame.jpg")
optical_flow = Image.open("optical_flow.jpg")

# Concatenate conditioning images (they will be concatenated in the pipeline)
# Note: You'll need to prepare the 6-channel input by concatenating prev_frame and optical_flow
prompt = "your prompt describing the scene"

# Generate
image = pipe(
    prompt=prompt,
    image=[prev_frame, optical_flow],  # The pipeline will handle concatenation
    num_inference_steps=20,
    guidance_scale=7.5
).images[0]

image.save("output.jpg")

Training Details

• Base Model: stabilityai/stable-diffusion-xl-base-1.0
• Training Resolution: Multi-resolution (512, 640, 768, 896, 1024px)
• Conditioning Channels: 6 (3 for previous frame + 3 for optical flow)
• Training Steps: 25,000
• Mixed Precision: bfloat16

Limitations

This model requires specific conditioning inputs:

1. The previous frame from your video sequence
2. The optical flow computed between frames

For best results, ensure your optical flow visualization uses a consistent color scheme and magnitude representation.

License

This model is released under the same license as SDXL (OpenRAIL++).