hybrid-sd-tinyvae-xl

🍰 Hybrid-sd-tinyvae-xl for Stable Diffusion (XL)

Hybrid-sd-tinyvae-xl is very tiny autoencoder which uses the same "latent API" as SDXL-VAE. Hybrid-sd-tinyvae-xl is a finetuned model based on the excellent work on TAESDXL. In general, we mainly fix the low-saturation problem encountering in SDXL base model, by which we strengthening the saturation and contrast of images to deliver more clarity and colorfulness. The model is useful for real-time previewing of the SDXL generation process. It saves 13x decoder inference time (61.2ms,fp16,V100) compared to using the SDXL decoder (802.7ms,fp16,V100), and you are very welcome to try it !!!!!!

T2I Comparison using one A100 GPU, The image order from left to right : SDXL-VAE -> TAESDXL -> Hybrid-sd-tinyvae-xl

This repo contains .safetensors versions of the Hybrid-sd-tinyvae-xl weights. For SD1.x / SD2.x, use Hybrid-sd-tinyvae instead (the SD and SDXL VAEs are incompatible).

Using in 🧨 diffusers

import torch
from diffusers.models import AutoencoderTiny
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
)

vae = AutoencoderTiny.from_pretrained('cqyan/hybrid-sd-tinyvae-xl')
pipe.vae = vae
pipe = pipe.to("cuda")
prompt = "A warm and loving family portrait, highly detailed, hyper-realistic, 8k resolution, photorealistic, soft and natural lighting"
image = pipe(prompt, num_inference_steps=25).images[0]
image.save("family.png")