AnySomniumAlpha

AnySomniumAlpha Model Teaser

<p align="center"> <img src="01.jpg" width=30% height=30%> </p>

Ketengan-Diffusion/AnySomniumAlpha is an experimental model that has been with pixart-α base model, fine-tuned from PixArt-alpha/PixArt-XL-2-1024-MS.

This is a first version of AnySomniumAlpha the first ever Anime style model in Pixart-α environment, there is still need a lot of improvement.

Our model use same dataset and curation as AnySomniumXL v2, but with better captioning. This model also support booru tag based caption and natural language caption.

How to Use this Model

Coming soon

Our Dataset Process Curation

<p align="center"> <img src="Curation.png" width=70% height=70%> </p>

Image source: Source1 Source2 Source3

Our dataset is scored using Pretrained CLIP+MLP Aesthetic Scoring model by https://github.com/christophschuhmann/improved-aesthetic-predictor, and We made adjusment into our script to detecting any text or watermark by utilizing OCR by pytesseract

This scoring method has scale between -1-100, we take the score threshold around 17 or 20 as minimum and 65-75 as maximum to pretain the 2D style of the dataset, Any images with text will returning -1 score. So any images with score below 17 or above 65 is deleted

The dataset curation proccess is using Nvidia T4 16GB Machine and takes about 7 days for curating 1.000.000 images.

Captioning process

We using combination of proprietary Multimodal LLM and open source multimodal LLM such as LLaVa 1.5 as the captioning process which is resulting more complex result than using normal BLIP2. Any detail like the clothes, atmosphere, situation, scene, place, gender, skin, and others is generated by LLM.

This captioning process to captioning 33k images takes about 3 Days with NVIDIA Tesla A100 80GB PCIe. We still improving our script to generate caption faster. The minimum VRAM that required for this captioning process is 24GB VRAM which is not sufficient if we using NVIDIA Tesla T4 16GB

Tagging Process

We simply using booru tags, that retrieved from booru boards so this could be tagged by manually by human hence make this tags more accurate.

Official Demo

Coming soon

Technical Specifications

AnySomniumAlpha Technical Specifications:

Batch Size: 8

Learning rate: 3e-6

Trained with a bucket size of 1024x1024

Datasets count: 33k Images

Text Encoder: t5-v1_1-xxl

Train datatype: tfloat32

Model weight: fp32

Trained with NVIDIA A100 80GB, Thanks to bilikpintar for computing resource for train AnySomniumAlpha

You can support me:

• on Ko-FI