weifu-diffusion-1-5-beta3-unofficial

WD 1.5 Beta 3 (Diffusers-compatible)

<img width="582px" height="256px" src="https://birchlabs.co.uk/share/radiance0triptych.jpg" title="Triptych of Reimu, Sanae and Flandre in 'radiance' aesthetic">

This unofficial repository hosts diffusers-compatible float16 checkpoints of WD 1.5 beta 3.
Float16 is all you need for inference.

NOTE: This version only hosts the base model as it is meant for finetuning with everydream2trainer (Whic does not support safetensors for some reason)

Usage (via diffusers)

# make sure you're logged in with `huggingface-cli login`
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
from diffusers.models.autoencoder_kl import AutoencoderKL
from diffusers.pipelines.stable_diffusion import StableDiffusionPipelineOutput
import torch
from torch import Generator, compile
from PIL import Image
from typing import List

vae: AutoencoderKL = AutoencoderKL.from_pretrained('hakurei/waifu-diffusion', subfolder='vae', torch_dtype=torch.float16)

# scheduler args documented here:
# https://github.com/huggingface/diffusers/blob/0392eceba8d42b24fcecc56b2cc1f4582dbefcc4/src/diffusers/schedulers/scheduling_dpmsolver_multistep.py#L83
scheduler: DPMSolverMultistepScheduler = DPMSolverMultistepScheduler.from_pretrained(
  'Birchlabs/wd-1-5-beta3-unofficial',
  subfolder='scheduler',
  # sde-dpmsolver++ is very new. if your diffusers version doesn't have it: use 'dpmsolver++' instead.
  algorithm_type='sde-dpmsolver++',
  solver_order=2,
  # solver_type='heun' may give a sharper image. Cheng Lu reckons midpoint is better.
  solver_type='midpoint',
  use_karras_sigmas=True,
)

# variant=None
# variant='ink'
# variant='mofu'
variant='radiance'
# variant='illusion'
pipe: StableDiffusionPipeline = StableDiffusionPipeline.from_pretrained(
  'Birchlabs/wd-1-5-beta3-unofficial',
  torch_dtype=torch.float16,
  vae=vae,
  scheduler=scheduler,
  variant=variant,
)
pipe.to('cuda')
compile(pipe.unet, mode='reduce-overhead')

# WD1.5 was trained on area=896**2 and no side longer than 1152
sqrt_area=896
# note: pipeline requires width and height to be multiples of 8
height = 1024
width = sqrt_area**2//height

prompt = 'artoria pendragon (fate), reddizen, 1girl, best aesthetic, best quality, blue dress, full body, white shirt, blonde hair, looking at viewer, hair between eyes, floating hair, green eyes, blue ribbon, long sleeves, juliet sleeves, light smile, hair ribbon, outdoors, painting (medium), traditional media'
negative_prompt = 'lowres, bad anatomy, bad hands, missing fingers, extra fingers, blurry, mutation, deformed face, ugly, bad proportions, monster, cropped, worst quality, jpeg, bad posture, long body, long neck, jpeg artifacts, deleted, bad aesthetic, realistic, real life, instagram'

# pipeline invocation args documented here:
# https://github.com/huggingface/diffusers/blob/0392eceba8d42b24fcecc56b2cc1f4582dbefcc4/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#LL544C18-L544C18
out: StableDiffusionPipelineOutput = pipe.__call__(
  prompt,
  negative_prompt=negative_prompt,
  height=height,
  width=width,
  num_inference_steps=22,
  generator=Generator().manual_seed(1234)
)
images: List[Image.Image] = out.images
img, *_ = images

img.save('out_pipe/saber.png')

Should output the following image:

<img height="256px" src="https://birchlabs.co.uk/share/saber-radiance.smol.jpg" title="Saber in 'radiance' aesthetic">

Original model card

For this release, we release five versions of the model:

• WD 1.5 Beta3 Base
• WD 1.5 Radiance
• WD 1.5 Ink
• WD 1.5 Mofu
• WD 1.5 Illusion

The WD 1.5 Base model is only intended for training use. For generation, it is recomended to create your own finetunes and loras on top of WD 1.5 Base or use one of the aesthetic models. More information and sample generations for the aesthetic models are in the release notes

Release Notes

https://saltacc.notion.site/WD-1-5-Beta-3-Release-Notes-1e35a0ed1bb24c5b93ec79c45c217f63

VAE

WD 1.5 uses the same VAE as WD 1.4, which can be found here https://huggingface.co/hakurei/waifu-diffusion-v1-4/blob/main/vae/kl-f8-anime2.ckpt

License

WD 1.5 is released under the Fair AI Public License 1.0-SD (https://freedevproject.org/faipl-1.0-sd/). If any derivative of this model is made, please share your changes accordingly. Special thanks to ronsor/undeleted (https://undeleted.ronsor.com/) for help with the license.