sdxl_lora_architecture_siheyuan

Overview

Architecture Lora Chinese Style is a lora training model with sdxl1.0 base model, latent text-to-image diffusion model. The model has been fine-tuned using a learning rate of 1e-5 over 3000 total steps with a batch size of 4 on a curated dataset of superior-quality chinese building style images. This model is derived from Stable Diffusion XL 1.0.

• Use it with 🧨 diffusers
• Use it with the ComfyUI (recommended)

Model Description

<!-- Provide a longer summary of what this model is. -->

• Developed by: FrankChieng
• Model type: Diffusion-based text-to-image generative model
• License: CreativeML Open RAIL++-M License
• Finetuned from model [optional]: Stable Diffusion XL 1.0 base

<hr>

How to Use:

• Download Lora model here, the model is in .safetensors format.
• You need to use include siheyuan prompt in natural language, then you will get realistic result image
• You can use any generic negative prompt or use the following suggested negative prompt to guide the model towards high aesthetic generationse:

low quality, low resolution,watermark, mark, nsfw, lowres, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark

• And, the following should also be prepended to prompts to get high aesthetic results:

masterpiece, best quality

<hr>

🧨 Diffusers

Make sure to upgrade diffusers to >= 0.18.2:

pip install diffusers --upgrade

In addition make sure to install transformers, safetensors, accelerate as well as the invisible watermark:

pip install invisible_watermark transformers accelerate safetensors

Running the pipeline (if you don't swap the scheduler it will run with the default EulerDiscreteScheduler in this example we are swapping it to EulerAncestralDiscreteScheduler:

pip install -q --upgrade diffusers invisible_watermark transformers accelerate safetensors
pip install huggingface_hub
from huggingface_hub import notebook_login
notebook_login()
import torch
from torch import autocast
from diffusers import StableDiffusionXLPipeline, EulerAncestralDiscreteScheduler

base_model_id = "stabilityai/stable-diffusion-xl-base-1.0"
lora_model = "frank-chieng/sdxl_lora_architecture_siheyuan"

pipe = StableDiffusionXLPipeline.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    use_safetensors=True,
    )
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights(lora_model, weight_name="sdxl_lora_architecture_siheyuan.safetensors")
pipe.to('cuda')
prompt = "siheyuan, chinese modern architecture, perfectly shaded, night lighting, medium closeup, mystical setting, during the day"
negative_prompt = "watermark"
image = pipe(
    prompt, 
    negative_prompt=negative_prompt, 
    width=1024,
    height=1024,
    guidance_scale=7,
    target_size=(1024,1024),
    original_size=(4096,4096),
    num_inference_steps=28
    ).images[0]
image.save("chinese_siheyuan.png")

<hr>

Limitation

This model inherit Stable Diffusion XL 1.0 limitation