procreate-diffusion-frank-gehry

ProCreate checkpoint: Frank Gehry

A Stable Diffusion 1.5 UNet fine-tuned for 2,000 steps on the Frank Gehry category of FSCG-8 (50 caption-image pairs of Frank Gehry architecture). It is one of eight checkpoints released with ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation (ECCV 2024).

ProCreate is a sampling method that pushes generated images away from the reference images, so a model fine-tuned on a few examples produces novel samples in the same style instead of replicating its training data.

Paper · Project page · Code · Dataset

What this repository contains

Only the fine-tuned UNet (UNet2DConditionModel, diffusers format). Load it into a Stable Diffusion 1.5 pipeline:

import torch
from diffusers import StableDiffusionPipeline, UNet2DConditionModel

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16
).to("cuda")
pipe.unet = UNet2DConditionModel.from_pretrained(
    "Jacklu0831/procreate-diffusion-frank-gehry", torch_dtype=torch.float16
).to("cuda")
image = pipe("a twisting tall apartment building, designed by Frank Gehry").images[0]

This samples from the fine-tuned model directly. To sample with ProCreate, use src/inference.py in the code repository, which downloads this checkpoint automatically:

python src/inference.py \
    --dataset_dir few-shot-creative-generation-8/frank_gehry \
    --unet_ckpt_dir Jacklu0831/procreate-diffusion-frank-gehry \
    --prompt "a twisting tall apartment building, designed by Frank Gehry"

License

The weights are a derivative of Stable Diffusion 1.5 and are released under the CreativeML Open RAIL-M license.

Citation

@InProceedings{procreate,
    author="Lu, Jack and Teehan, Ryan and Ren, Mengye",
    title="ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generation",
    booktitle="Computer Vision -- ECCV 2024",
    year="2024",
    publisher="Springer Nature Switzerland",
}