TokenCompose_SD14_A

🧩 TokenCompose SD14 Model Card

🎬CVPR 2024

TokenCompose_SD14_A is a latent text-to-image diffusion model finetuned from the Stable-Diffusion-v1-4 checkpoint at resolution 512x512 on the VSR split of COCO image-caption pairs for 24,000 steps with a learning rate of 5e-6. The training objective involves token-level grounding terms in addition to denoising loss for enhanced multi-category instance composition and photorealism. The "_A/B" postfix indicates different finetuning runs of the model using the same above configurations.

📄 Paper

Please follow this link.

🧨Example Usage

We strongly recommend using the 🤗Diffuser library to run our model.

import torch
from diffusers import StableDiffusionPipeline

model_id = "mlpc-lab/TokenCompose_SD14_A"
device = "cuda"

pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)
pipe = pipe.to(device)

prompt = "A cat and a wine glass"
image = pipe(prompt).images[0]  
    
image.save("cat_and_wine_glass.png")

⬆️Improvements over SD14

<table> <tr> <th rowspan="3" align="center">Method</th> <th colspan="9" align="center">Multi-category Instance Composition</th> <th colspan="2" align="center">Photorealism</th> <th colspan="1" align="center">Efficiency</th> </tr> <tr> <!-- <th align="center">&nbsp;</th> --> <th rowspan="2" align="center">Object Accuracy</th> <th colspan="4" align="center">COCO</th> <th colspan="4" align="center">ADE20K</th> <th rowspan="2" align="center">FID (COCO)</th> <th rowspan="2" align="center">FID (Flickr30K)</th> <th rowspan="2" align="center">Latency</th> </tr> <tr> <!-- <th align="center">&nbsp;</th> --> <th align="center">MG2</th> <th align="center">MG3</th> <th align="center">MG4</th> <th align="center">MG5</th> <th align="center">MG2</th> <th align="center">MG3</th> <th align="center">MG4</th> <th align="center">MG5</th> </tr> <tr> <td align="center"><a href="https://huggingface.co/CompVis/stable-diffusion-v1-4">SD 1.4</a></td> <td align="center">29.86</td> <td align="center">90.72_1.33</td> <td align="center">50.74_0.89</td> <td align="center">11.68_0.45</td> <td align="center">0.88_0.21</td> <td align="center">89.81_0.40</td> <td align="center">53.96_1.14</td> <td align="center">16.52_1.13</td> <td align="center">1.89_0.34</td> <td align="center"><u>20.88</u></td> <td align="center"><u>71.46</u></td> <td align="center"><b>7.54</b>_0.17</td> </tr> <tr> <td align="center"><a href="https://github.com/mlpc-ucsd/TokenCompose"><strong>TokenCompose (Ours)</strong></a></td> <td align="center"><b>52.15</b></td> <td align="center"><b>98.08</b>_0.40</td> <td align="center"><b>76.16</b>_1.04</td> <td align="center"><b>28.81</b>_0.95</td> <td align="center"><u>3.28</u>_0.48</td> <td align="center"><b>97.75</b>_0.34</td> <td align="center"><b>76.93</b>_1.09</td> <td align="center"><b>33.92</b>_1.47</td> <td align="center"><b>6.21</b>_0.62</td> <td align="center"><b>20.19</b></td> <td align="center"><b>71.13</b></td> <td align="center"><b>7.56</b>_0.14</td> </tr> </table>

📰 Citation

@InProceedings{Wang2024TokenCompose,
    author    = {Wang, Zirui and Sha, Zhizhou and Ding, Zheng and Wang, Yilin and Tu, Zhuowen},
    title     = {TokenCompose: Text-to-Image Diffusion with Token-level Supervision},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {8553-8564}
}