sdxl-base-1.0-jarekl-lora

LoRA Weights for sdxl-base-1.0 Tuned on Jarek Lucek's Photos

These are LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0. The weights were fine-tuned on the AdamLucek/jarekl-photos dataset.

Explicit permissions given for training from Jarek Lucek, more information in dataset.

Example Images:

"Evening cityscape highlighting the interplay of light on traditional architecture, stairs, and fencing, with a distant human figure for scale." <img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/5xLXUDfnf4JK2ntZ0pSiO.png" width="300">

"Night scene of a quiet urban street, featuring architectural details of a prominent building and a single person in the foreground." <img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/1k1M5L6Q8RPHTdTeZ-oXR.png" width="300">

"An old church stands tall in the darkness, its white walls and steeple glowing under streetlights, with a solitary pedestrian passing by." <img src="https://cdn-uploads.huggingface.co/production/uploads/65ba68a15d2ef0a4b2c892b4/6f9o_giTqpKq4QDy5nshs.png" width="300">

Intended uses & limitations

Intended uses

These LoRA weights are designed solely for educational and research purposes, intended to replicate the aesthetic and technique characteristic of Jarek Lucek's photography style. By using this model, you acknowledge that the generated images are simulations and do not reflect the original works directly.

The images produced by this model are NOT authorized for commercial use. They may not be sold, licensed, or otherwise exploited for any commercial purpose without explicit permission from Jarek Lucek.

Users must attribute the generated images directly to Jarek Lucek. Users are advised to ensure their use complies with applicable laws and respects the artistic rights of the original creator.

By downloading or using this model, you agree to abide by these terms and accept full responsibility for the use of the generated images.

How to use

Importing SDXL with LoRA & Refiner

from diffusers import DiffusionPipeline
import torch

base_sdxl = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", 
    torch_dtype=torch.float16, 
    variant="fp16", 
    use_safetensors=True
).to("cuda")
base_sdxl.enable_model_cpu_offload()

base_sdxl.load_lora_weights("AdamLucek/sdxl-base-1.0-jarekl-lora")

refiner = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    text_encoder_2=base_sdxl.text_encoder_2,
    vae=base_sdxl.vae,
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
).to("cuda")

Generating an Image

prompt = "Evening cityscape highlighting the interplay of light on traditional architecture, stairs, and fencing, with a distant human figure for scale."

image = base_sdxl(
    prompt=prompt,
    num_inference_steps=50,
    denoising_end=0.8,
    output_type="latent",
).images

image = refiner(
    prompt=prompt,
    num_inference_steps=50,
    denoising_start=0.8,
    image=image,
).images[0]

image.save("cityscape.png")

Limitations and bias

Note: First pass on LoRA training, not refined or tuned specifically.

Note: Hyperparameters are not very scientifically chosen as this is a first attempt.

Training details

Trained using a single a10 using diffusers package, documentation available here.

Training Script:

accelerate launch train_text_to_image_lora_sdxl.py \
 --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
 --dataset_name=AdamLucek/jarekl-photos \
 --output_dir=output/sdxl-base-1.0-jarekl-lora \
 --resolution=1024 \
 --train_batch_size=1 \
 --dataloader_num_workers=8 \
 --gradient_accumulation_steps=4 \
 --max_train_steps=2000 \
 --num_train_epochs=50 \
 --learning_rate=1e-04 \
 --random_flip \
 --max_grad_norm=1 \
 --lr_scheduler="cosine" \
 --lr_warmup_steps=100  \
 --use_8bit_adam \
 --allow_tf32 \
 --checkpointing_steps=100 \
 --validation_prompt="Black and white photograph of a narrow cobblestone street at night. A lone figure stands in the foreground, looking into a shop window. The street is lined with old buildings, their textures emphasized by the monochrome image. In the background, a church spire is visible against the night sky. Street lamps cast light pools on the cobblestones, creating a moody scene." \
 --num_validation_images=4 \
 --validation_epochs=1 \
 --mixed_precision="fp16" \
 --seed=66 \
 --push_to_hub \
 --hub_model_id="sdxl-base-1.0-jarekl-lora"