general-finetuned-mb

Paint by Inpaint: Learning to Add Image Objects by Removing Them First

The model is designed for instruction-following object addition to images.

We offer four different models:

• Trained on the PIPE dataset, specifically designed for object addition.
• The object addition model fine-tuned on a MagicBrush addition subset.
• Trained on the combined PIPE and InstructPix2Pix datasets, intended for general editing.
• The general model fine-tuned on the full MagicBrush dataset (This one).

Resources

• 💻 Visit Project Page

• 📝 Read the Paper

• 🚀 Try Our Demo

• 🗂️ Use PIPE Dataset

Running the model

The model is simple to run using the InstructPix2Pix pipeline:

from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler
import torch
import requests
from io import BytesIO

model_name = "paint-by-inpaint/general-finetuned-mb"

diffusion_steps = 50
device = "cuda"
image_url = "https://paint-by-inpaint-demo.hf.space/file=/tmp/gradio/99cd3a15aa9bdd3220b4063ebc3ac05e07a611b8/messi.jpeg"
image = Image.open(BytesIO(requests.get(image_url).content)).resize((512, 512))

pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_name, torch_dtype=torch.float16, safety_checker=None).to(device)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

# Generate the modified image
out_images = pipe(
    "Add a royal silver crown", 
    image=image, 
    guidance_scale=7, 
    image_guidance_scale=1.5, 
    num_inference_steps=diffusion_steps,
    num_images_per_prompt=1
).images

BibTeX

@article{wasserman2024paint,
  title={Paint by Inpaint: Learning to Add Image Objects by Removing Them First},
  author={Wasserman, Navve and Rotstein, Noam and Ganz, Roy and Kimmel, Ron},
  journal={arXiv preprint arXiv:2404.18212},
  year={2024}
}