TIPO-500M-GGUF

QuantFactory/TIPO-500M-GGUF

This is quantized version of KBlueLeaf/TIPO-500M created using llama.cpp

Original Model Card

TIPO: Text to Image with text presampling for Prompt Optimization

500M LLaMA arch model trained for TIPO.<br> Tech Report: https://hackmd.io/@KBlueLeaf/BJULOQBR0

Introduction

In this project, we introduce "TIPO" (Text to Image with text presampling for Prompt Optimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.

Usage

Use updated version of DTG extension (renamed to z-tipo-extension), current version of z-tipo-extension support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested. https://github.com/KohakuBlueleaf/z-tipo-extension

Model arch and Training

This model is LLaMA arch with 500M parameters, the training data is combined version of Danbooru2023, GBC10M and Coyo-HD-11M.<br> The total token seen is around 30B tokens.<br> For more information please refer to the tech report and following table.

*: We only count "non-padding token" in the token seen, since all the training data have very large length range <br/> `: Since the training data is pretty short, it cost more time to reach same token seen than general LLM pretraining.<br/> As reference, with 4096 as max ctx length and almost all the data have reach that length, you may only need 2days to reach 10B token seen on RTX 3090 x 4 with 200M model.

Evaluation

We have tested TIPO in several metric:

1. Aesthetic Score (Higher is Better)

We compute the Aesthetic Score using the Aesthetic Predictor V2.5. This metric is calculated on the short/truncated long test.

Figure 1: Aesthetic Score distribution.

2. AI Corrupt Score (Higher is Better)

The AI Corrupt Score is obtained from the AICorruptMetrics in sdeval.

This metric is calculated on the short/truncated long test.

Figure 2: AI Corrupt Score distribution.

3. Frechet Dino Distance (FDD) on Scenery Tag Test

We use FDD on the Scenery Tag Test to demonstrate that when input prompts address a smaller distribution, the model struggles to generate images that reflect the true distribution. However, with TIPO, this issue is mitigated.

Table 1: Frechet Dino Distance (FDD) on Scenery Tag Test.

LICENSE

This model is released under Kohaku License 1.0<br> You can check the above provided URL or check the LICENSE file in this repo.

Citation

@misc{yeh2024tipo,
  title = {TIPO: Text to Image with text presampling for Prompt Optimization},
  author = {Yeh, Shih-Ying},
  year = {2024},
  month = {9},
  day = {29},
  note = {Technical report available at \url{https://hackmd.io/@KBlueLeaf/BJULOQBR0}. 
          Model available at \url{https://huggingface.co/KBlueLeaf/TIPO-500M}. 
          Source code available at \url{https://github.com/KohakuBlueleaf/KGen}},
}