longt5-stable-diffusion-prompt

Model description

This model is a fine-tuned version of pszemraj/long-t5-tglobal-base-16384-book-summary on a custom sample-size dataset. The dataset was kmfoda/booksum fed into GPT3.5-turbo with a finely tuned prompt to output high quality Stable Diffusion prompts. The small dataset (less than $10 of OpenAI credits) was roughly 15k entries as a proof of concept.

The goal for this model concept was to create a text summarization model that creates decent Stable Diffusion prompts comparable to a human or high-end LLM like GPT-4.

Example generations from an excerpt of Hemingway:

this model: village in late summer, river and plain, mountains, pebbled boulders, blue water, troops marching, dusty trees, soldiers marching along road, crops rich with fruit trees, battle in the mountains, artillery flashes, cool nights, highly detailed, dramatic lighting

gpt-4: desert landscape with camel caravan at sunset, nomad tents, sand dunes, oasis, traditional clothing, dramatic lighting, 8k UHD, highly detailed, masterpiece, digital painting, global illumination

This is a VERY rough proof-of-concept model that could be greatly improved by a higher quality dataset and possibly different hyperparameters.

Training procedure

Training was completed over 7 epochs with a modified version of the run_summarization.py Huggingface training script.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 6
• total_train_batch_size: 48
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 7.0

Training results

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.1+cu118
• Datasets 2.15.0
• Tokenizers 0.15.0