sd35m-photo-SLG-shift0.8-attn_ff-512to2048px
This is a standard PEFT LoRA derived from stabilityai/stable-diffusion-3.5-medium.
The main validation prompt used during training was:
A photo-realistic image of a cat
Validation settings
- CFG:
6.0
- CFG Rescale:
0.0
- Steps:
30
- Sampler:
FlowMatchEulerDiscreteScheduler
- Seed:
42
- Resolution:
1024x1024
- Skip-layer guidance:
skip_guidance_layers=[7, 8, 9],
Note: The validation settings are not necessarily the same as the training settings.
You can find some example images in the following gallery:
<Gallery />
The text encoder was not trained.
You may reuse the base model text encoder for inference.
Training settings
-
Training epochs: 1
-
Training steps: 64000
-
Learning rate: 0.0003
- Learning rate schedule: constant
- Warmup steps: 500
-
Max grad norm: 0.01
-
Effective batch size: 9
- Micro-batch size: 3
- Gradient accumulation steps: 1
- Number of GPUs: 3
-
Gradient checkpointing: True
-
Prediction type: flow-matching (extra parameters=['flux_schedule_auto_shift', 'shift=0.0', 'flux_use_uniform_schedule'])
-
Optimizer: adamw_bf16
-
Trainable parameter precision: Pure BF16
-
Caption dropout probability: 25.0%
-
LoRA Rank: 128
-
LoRA Alpha: 128.0
-
LoRA Dropout: 0.1
-
LoRA initialisation style: default
Datasets
signs
- Repeats: 0
- Total number of images: ~420
- Total number of aspect buckets: 14
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
moviecollection
- Repeats: 0
- Total number of images: ~1983
- Total number of aspect buckets: 34
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
bookcovers
- Repeats: 0
- Total number of images: ~927
- Total number of aspect buckets: 26
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
shutterstock
- Repeats: 0
- Total number of images: ~21111
- Total number of aspect buckets: 33
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
cinemamix-1mp
- Repeats: 0
- Total number of images: ~7425
- Total number of aspect buckets: 5
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
anatomy
- Repeats: 5
- Total number of images: ~16440
- Total number of aspect buckets: 24
- Resolution: 1.048576 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
signs-512
- Repeats: 0
- Total number of images: ~417
- Total number of aspect buckets: 12
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
moviecollection-512
- Repeats: 0
- Total number of images: ~1971
- Total number of aspect buckets: 19
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
bookcovers-512
- Repeats: 0
- Total number of images: ~918
- Total number of aspect buckets: 16
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
shutterstock-512
- Repeats: 0
- Total number of images: ~21096
- Total number of aspect buckets: 19
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
cinemamix-1mp-512
- Repeats: 0
- Total number of images: ~7422
- Total number of aspect buckets: 3
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
anatomy-512
- Repeats: 5
- Total number of images: ~16437
- Total number of aspect buckets: 13
- Resolution: 0.262144 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
signs-1440
- Repeats: 100
- Total number of images: ~423
- Total number of aspect buckets: 1
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
moviecollection-1440
- Repeats: 0
- Total number of images: ~2007
- Total number of aspect buckets: 44
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
bookcovers-1440
- Repeats: 0
- Total number of images: ~933
- Total number of aspect buckets: 29
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
shutterstock-1440
- Repeats: 0
- Total number of images: ~21111
- Total number of aspect buckets: 42
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
cinemamix-1mp-1440
- Repeats: 0
- Total number of images: ~7425
- Total number of aspect buckets: 5
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
anatomy-1440
- Repeats: 5
- Total number of images: ~16458
- Total number of aspect buckets: 32
- Resolution: 2.0736 megapixels
- Cropped: False
- Crop style: None
- Crop aspect: None
- Used for regularisation data: No
Inference
import torch
from diffusers import DiffusionPipeline
model_id = 'stabilityai/stable-diffusion-3.5-medium'
adapter_id = 'bghira/sd35m-photo-SLG-shift0.8-attn_ff-512to2048px'
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
pipeline.load_lora_weights(adapter_id)
prompt = "A photo-realistic image of a cat"
negative_prompt = 'ugly, cropped, blurry, low-quality, mediocre average'
## Optional: quantise the model to save on vram.
## Note: The model was not quantised during training, so it is not necessary to quantise it during inference time.
#from optimum.quanto import quantize, freeze, qint8
#quantize(pipeline.transformer, weights=qint8)
#freeze(pipeline.transformer)
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=30,
generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
width=1024,
height=1024,
guidance_scale=6.0,
skip_guidance_layers=[7, 8, 9],
).images[0]
image.save("output.png", format="PNG")
Exponential Moving Average (EMA)
SimpleTuner generates a safetensors variant of the EMA weights and a pt file.
The safetensors file is intended to be used for inference, and the pt file is for continuing finetuning.
The EMA model may provide a more well-rounded result, but typically will feel undertrained compared to the full model as it is a running decayed average of the model weights.