sd-2-inpainting_background_lora_v1

Intended uses & limitations

• Model usage: LoRA model for stable-inpainting-2 model to generate various background types: abstract, cartoon, forest
• This is not a dreambooth finetuning process, LoRA finetuning on UNet component
• Random mask were created during the training process

How to use

from diffusers import DiffusionPipeline
import torch
from safetensors.torch import load_file

# Load the pre-trained model
pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-inpainting", torch_dtype=torch.float16)
pipe = pipe.to("cuda")  # Move the model to GPU for faster processing

lora_model_path = "pytorch_lora_weights.safetensors"

# Load and apply LoRA weights
state_dict = load_file(lora_model_path)
pipe.unet.load_attn_procs(state_dict)

prompt = "A farm in the background with flowers, warm lighting, and a cozy, scholarly atmosphere."
negative_prompt = "deformed, ugly, disfigured"

# Generate a new image
generator = torch.Generator(device='cuda:0').manual_seed(123456)
image = pipe(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator, guidance_scale=5, strength=1).images[0]

Generated Examples

Generated prompt: a dark jungle, gloomy sky

Generated prompt: Create a cozy living room setting with warm lighting and soft furnishings, placing the pet bed near a fireplace for comfort. Use neutral or warm colours such as beige or light brown for the background to complement the bed's colour and material

Generated prompt: Create a calming living space with a natural color palette and soft lighting, evoking tranquility and comfort. Subtle, diffused light and earthy tones blend seamlessly, forging a cozy atmosphere.

Training details

• No.of steps the model was trained: 4000 steps
• Batch-size: 16
• No.of images used: 1,000
• Masking strategy: random mask
• LoRA rank: 256