sdxl_smoothed_energy_guidance

Smoothed Energy Guidance for SDXL

https://arxiv.org/abs/2408.00760 | https://colab.research.google.com/github/SusungHong/SEG-SDXL/blob/master/sdxl_seg.ipynb

Identical to https://github.com/SusungHong/SEG-SDXL/blob/8d3b2007a5f0660f9dba110a5e83556395f7535f/pipeline_seg.py

Implementation of Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attention by Susung Hong.

<img src="./teaser-2.jpg" width="90%">

🏔️ What is Smoothed Energy Guidance? How does it work?

Smoothed Energy Guidance (SEG) is a training- and condition-free approach that leverages the energy-based perspective of the self-attention mechanism to improve image generation.

Key points:

• Does not rely on the guidance scale parameter that causes side effects when the value becomes large
• Allows continuous control of the original and maximally attenuated curvature of the energy landscape behind self-attention
• Introduces a query blurring method, equivalent to blurring the entire attention weights without significant computational cost

🔍 Comparison with other works

SEG does not severely suffer from side effects such as making the overall image grayish or significantly changing the original structure, while improving generation quality even without prompts.

Unconditional generation without prompts <img src="./seg_comparison-2.jpg" width="90%">