sdxl_perturbed_attention_guidance

Perturbed-Attention Guidance for SDXL

<div style="display:flex"> <video width=50% autoplay loop controls> <source src="https://huggingface.co/multimodalart/sdxl_perturbed_attention_guidance/resolve/main/pag_sdxl.mp4" type="video/mp4"> </video> <video width=50% autoplay loop controls> <source src="https://huggingface.co/multimodalart/sdxl_perturbed_attention_guidance/resolve/main/pag_uncond.mp4" type="video/mp4"> </video> </div>

The original Perturbed-Attention Guidance for unconditional models and SD1.5 by Hyoungwon Cho is availiable at hyoungwoncho/sd_perturbed_attention_guidance

Project / arXiv / GitHub

This repository is just a simple SDXL implementation of the Perturbed-Attention Guidance (PAG) on Stable Diffusion XL (SDXL) for the 🧨 diffusers library.

Quickstart

Loading Custom Pipeline:

from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    custom_pipeline="multimodalart/sdxl_perturbed_attention_guidance",
    torch_dtype=torch.float16
)

device="cuda"
pipe = pipe.to(device)

Unconditional sampling with PAG:

output = pipe(
        "",
        num_inference_steps=50,
        guidance_scale=0.0,
        pag_scale=5.0,
        pag_applied_layers=['mid']
    ).images

Sampling with PAG and CFG:

output = pipe(
        "the spirit of a tamagotchi wandering in the city of Vienna",
        num_inference_steps=25,
        guidance_scale=4.0,
        pag_scale=3.0,
        pag_applied_layers=['mid']
    ).images

Parameters

guidance_scale : guidance scale of CFG (ex: 7.5)

pag_scale : guidance scale of PAG (ex: 4.0)

pag_applied_layers: layer to apply perturbation (ex: ['mid'])

pag_applied_layers_index : index of the layers to apply perturbation (ex: ['m0', 'm1'])

Stable Diffusion XL Demo

Try it here