TIPO-200M-ft

TIPO: Text to Image with text presampling for Prompt Optimization

200M LLaMA arch model trained for TIPO. <br> Tech Report: https://arxiv.org/abs/2411.08127

Introduction

In this project, we introduce "TIPO" (Text to Image with text presampling for Prompt Optimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.

Usage

Use updated version of DTG extension (renamed to z-tipo-extension), current version of z-tipo-extension support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested. https://github.com/KohakuBlueleaf/z-tipo-extension

Model arch and Training

This model is LLaMA arch with 200M parameters, the training data is combined version of Danbooru2023, Coyo-HD-11M. <br> The total token seen is around 50B tokens. <br> For more information please refer to the tech report and following table.

*: We only count "non-padding token" in the token seen, since all the training data have very large length range. <br> `: Since the training data is pretty short, it cost more time to reach same token seen than general LLM pretraining. <br> As reference, with 4096 as max ctx length and almost all the data have reach that length, you may only need 2days to reach 10B token seen on RTX 3090 x 4 with 200M model.

Evaluation

Evaluation are done on TIPO-200M model <br> We have tested TIPO compared to other Model in several test and metrics:

Scenery tag test

In this test we use single "scenery" tag as input. (With some certain meta) <br> To test each prompt gen method to see if they can obtain the desired distribution of outputs while maintain the quality of images.

Short/Truncated Long test

In this test we use short caption or manually truncated caption from GBC10M and CoyoHD11M. <br> This test examine the ability of prompt gen method on handling almostly completed prompts.

LICENSE

This model is released under Kohaku License 1.0 <br> You can check the above provided URL or check the LICENSE file in this repo.

Citation

@misc{yeh2024tipotextimagetext,
      title={TIPO: Text to Image with Text Presampling for Prompt Optimization}, 
      author={Shih-Ying Yeh and Sang-Hyun Park and Giyeong Oh and Min Song and Youngjae Yu},
      year={2024},
      eprint={2411.08127},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.08127}, 
}