dreambooth_diffusion_model-v2

Model description

This is a Stable Diffusion model fine-tuned using Dreambooth on Calvin and Hobbes images 🐯. Part of the Keras Dreambooth Event. An improvement on Original Dreambooth - CalHobbes using 50 images and 11 training epochs (V2) versus V1 (10 images and 3 training epochs).

Intended uses & limitations

• For experimentation and curiosity.

Training and evaluation data

• This model was fine-tuned on images of Calvin and Hobbes.

Training procedure

Starting from the provided Keras Dreambooth Sprity - HuggingFace - GITHUB, the provided IPYNB was modified to accomidate user images and optimze for cost. The entire training process was done using Colab. Data preperation can complete using the free tier, but you will need a premium GPU (A100 - 40G) to train. Generate the images for class-images using free tier Colab, this will take 2-3 hours for 300 images. Once complete, you have 2 options, create the folder/root/.keras/datasets/class-images and copy the images to that directory, or create a tar.gz file and download the files for backup using the snippit below. Downloading the file will save you the compute of having to redo it again as it can simply be uploaded into the contents folder at a later time. You can collect the images for my_images into a folder in either PNG or JPG format, and upload the TAR.GZ version of the folder containing the images.

import glob
import tarfile

output_filename = "class-images.tar.gz"
with tarfile.open(output_filename, "w:gz") as tar:
	for file in glob.glob('class-images/*'):
		tar.write(file)

To use tf.keras.util.get_file with a tar.gz file located on your running VM you will need instance_images_root = tf.keras.utils.get_file(origin="file:///LOCATION_TO_TARGZ_FILE/my_images.tar.gz",untar=True) get_util - Tensorflow Docs. The command, in my case, places the files in '/root/.keras/datasets/my_images'. This will not work. The following work around resolved the issue:

# From within the Colab Notebook
!mkdir /root/.keras/datasets/class-images
!mkdir /root/.keras/datasets/my_images
!cp /root/.keras/datasets/my_images.tar.gz /root/.keras/datasets/my_images/
!cp /root/.keras/datasets/class-images.tar.gz /root/.keras/datasets/class-images/
!tar -xvzf /root/.keras/datasets/class-images/class-images.tar.gz -C /root/.keras/datasets/class-images
!tar -xvzf /root/.keras/datasets/my_images/my_images.tar.gz -C /root/.keras/datasets/my_images

Having more my_images will improve the results. Current results used 10, but 20-30 is recommended based on Implementation of DreamBooth using KerasCV and TensorFlow - Notes on preparing data for DreamBooth training of faces.

If you decide to train using Lambda GPU - Dreambooth - GIT be sure the python -m pip install tensorflow==2.11 instead of python -m pip install tensorflow as 2.12 will be downloaded by default and there will be a CUDA version mismatch, causing computation to occur on the CPU instead of the GPU. A second note is setting export XLA_FLAGS=--xla_gpu_cuda_data_dir=/usr/lib/cuda and export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/ from a terminal will not transfer to a notebook. Be sure to set the environment variables in the jupyter notbook to enable the proper CUDA libraries to be used otherwise the GPU will not be accessable. %env MY_ENV_VAR=value can help and must be the first code block of the notebook.

Training results

<details> <summary>View Inference Images - V1</summary>

</details> <details> <summary>View Inference Images - V2</summary>

</details>

Training hyperparameters

The following hyperparameters were used during training:

Recommendations

• Access to A100 GPUs can be a challenege. A variety of providers, at the time of this writing (April 2023), either do not have additional resources or may be excessivly expensive. Spot instances, or breaking the data prep and training onto different types of machines can alleviate this issue.

Environmental Impact

Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). In total roughly 10 hours were used primarily in US Central, with AWS as the reference. Additional resources are available at Our World in Data - CO2 Emissions

• Hardware Type: Intel Xeon (12 core VM), 85GB RAM, 80GB Disk, with NVIDIA A100-SXM 40GB
• Hours Used: 10 hrs
• Cloud Provider: Colab Free + Colab Premium
• Compute Region: US Central
• Carbon Emitted: 1.42 kg (GPU) + 0.59 kg (CPU) = 2 kg (the weight of 2 liters of water) - 2 kg offset = 0

Model Plot

<details> <summary>View Model Plot</summary>

</details>

Citation

• Diffusers - HuggingFace - GITHUB
• Model Card - HuggingFace Hub - GITHUB

@online{ruizDreamBoothFineTuning2023,
  title = {{{DreamBooth}}: {{Fine Tuning Text-to-Image Diffusion Models}} for {{Subject-Driven Generation}}},
  shorttitle = {{{DreamBooth}}},
  author = {Ruiz, Nataniel and Li, Yuanzhen and Jampani, Varun and Pritch, Yael and Rubinstein, Michael and Aberman, Kfir},
  date = {2023-03-15},
  number = {arXiv:2208.12242},
  eprint = {arXiv:2208.12242},
  eprinttype = {arxiv},
  url = {http://arxiv.org/abs/2208.12242},
  urldate = {2023-03-26},
  abstract = {Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/},
  pubstate = {preprint}
}

@article{owidco2andothergreenhousegasemissions,
    author = {Hannah Ritchie and Max Roser and Pablo Rosado},
    title = {CO₂ and Greenhouse Gas Emissions},
    journal = {Our World in Data},
    year = {2020},
    note = {https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions}
}

@article{lacoste2019quantifying,
  title={Quantifying the Carbon Emissions of Machine Learning},
  author={Lacoste, Alexandre and Luccioni, Alexandra and Schmidt, Victor and Dandres, Thomas},
  journal={arXiv preprint arXiv:1910.09700},
  year={2019}
}