sdxl-fp8-loras

<!-- README Version: v1.5 -->

SDXL FP8 LoRAs Collection

A curated collection of Low-Rank Adaptation (LoRA) models optimized for Stable Diffusion XL (SDXL) in FP8 precision format. LoRAs enable efficient fine-tuning and style adaptation for SDXL models with minimal disk space and memory requirements.

Model Description

This repository contains LoRA adapters for SDXL models that can modify and enhance image generation with specific styles, concepts, or characteristics. LoRAs work by applying learned modifications to the base SDXL model's attention layers, enabling:

• Style Transfer: Apply artistic styles (anime, photorealistic, painterly, etc.)
• Character/Subject Training: Generate specific characters, faces, or objects
• Concept Learning: Teach the model new concepts not in the original training
• Quality Enhancement: Improve details, lighting, composition, or specific aspects
• Efficiency: Much smaller than full models (typically 10-200MB vs 6.5GB)

Key Features

• FP8 Precision: Optimized 8-bit floating point format for reduced memory usage
• Stackable: Multiple LoRAs can be combined for complex effects
• Adjustable Strength: Control LoRA influence with weight parameters (0.0-1.0+)
• Fast Loading: Quick adapter switching without reloading base model
• Minimal VRAM: Add styles with negligible memory overhead

Repository Contents

Current Structure:

E:\huggingface\sdxl-fp8-loras\
├── README.md              # This file
└── loras\
    └── sdxl\              # SDXL LoRA models directory (awaiting models)

Expected LoRA File Formats:

• .safetensors - Primary secure format for LoRA weights
• .ckpt / .pt - Legacy PyTorch checkpoint formats (less common)

Typical LoRA Sizes:

• Small LoRAs (rank 8-16): 10-50 MB
• Medium LoRAs (rank 32-64): 50-150 MB
• Large LoRAs (rank 128+): 150-300 MB

Hardware Requirements

Minimum Requirements

• VRAM: Same as base SDXL model (8GB+ recommended)
• RAM: 16GB system RAM
• Disk Space: 10-300 MB per LoRA model
• GPU: NVIDIA GPU with CUDA support (RTX 20/30/40 series recommended)

Recommended Requirements

• VRAM: 12GB+ for multiple LoRA stacking
• RAM: 32GB for comfortable workflow
• Disk Space: 5-10GB for LoRA collection
• GPU: RTX 3080/4070 or better for fast generation

Performance Notes

• LoRAs add minimal inference overhead (typically <5%)
• Multiple LoRAs (2-4) can be stacked with minor slowdown
• FP8 precision maintains quality while reducing memory usage
• Loading/unloading LoRAs is much faster than switching full models

Usage Examples

Using with Diffusers (Python)

from diffusers import DiffusionPipeline
import torch

# Load base SDXL model
pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
).to("cuda")

# Load single LoRA
pipe.load_lora_weights(
    r"E:\huggingface\sdxl-fp8-loras\loras\sdxl",
    weight_name="your_lora_name.safetensors"
)

# Generate image with LoRA
prompt = "a beautiful landscape, detailed, high quality"
image = pipe(
    prompt,
    num_inference_steps=30,
    guidance_scale=7.5
).images[0]

image.save("output.png")

Stacking Multiple LoRAs

from diffusers import DiffusionPipeline
import torch

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")

# Load multiple LoRAs with different weights
pipe.load_lora_weights(
    r"E:\huggingface\sdxl-fp8-loras\loras\sdxl",
    weight_name="style_lora.safetensors",
    adapter_name="style"
)
pipe.load_lora_weights(
    r"E:\huggingface\sdxl-fp8-loras\loras\sdxl",
    weight_name="detail_lora.safetensors",
    adapter_name="detail"
)

# Set LoRA weights (0.0-1.0+ range, adjust for desired effect)
pipe.set_adapters(["style", "detail"], adapter_weights=[0.8, 0.6])

prompt = "a portrait, intricate details, artistic style"
image = pipe(prompt, num_inference_steps=30).images[0]
image.save("stacked_loras.png")

Adjusting LoRA Strength

# Lower weight (0.3-0.5): Subtle effect
pipe.load_lora_weights(path, weight_name="lora.safetensors")
pipe.fuse_lora(lora_scale=0.4)  # Subtle influence

# Medium weight (0.6-0.8): Balanced effect
pipe.fuse_lora(lora_scale=0.7)  # Standard strength

# Higher weight (0.9-1.2): Strong effect
pipe.fuse_lora(lora_scale=1.0)  # Maximum intended effect

Using with ComfyUI

1. Place LoRA files in ComfyUI's models/loras/ directory
2. In workflow, add "Load LoRA" node
3. Connect to model chain before KSampler
4. Set LoRA strength (model_strength and clip_strength parameters)
5. Generate with adjusted prompt for LoRA style

Using with Automatic1111 WebUI

1. Place LoRA files in stable-diffusion-webui/models/Lora/
2. In prompt, use syntax: <lora:filename:weight>
3. Example: a castle <lora:fantasy_style:0.8>
4. Adjust weight value to control LoRA influence
5. Generate image normally

Model Specifications

Architecture

• Base Model: Stable Diffusion XL (SDXL) 1.0
• Adapter Type: Low-Rank Adaptation (LoRA)
• Precision: FP8 (8-bit floating point)
• Format: SafeTensors (primary)
• Typical Ranks: 8, 16, 32, 64, 128 (higher = more parameters)

LoRA Technical Details

• Layer Targeting: Usually attention layers (Q, K, V projections)
• Parameter Efficiency: 0.1-5% of base model parameters
• Training Method: Fine-tuning on specific datasets/styles
• Compatibility: Works with SDXL base and refiner models

Quality Considerations

• FP8 precision maintains ≥95% quality of FP16 LoRAs
• Minimal quality loss compared to full model fine-tuning
• Stackability allows complex style combinations
• Weight adjustment enables fine-grained control

Performance Tips and Optimization

LoRA Selection Strategy

• Start Simple: Test single LoRAs before stacking
• Check Compatibility: Some LoRAs may conflict when stacked
• Weight Experimentation: Adjust weights between 0.3-1.2 for best results
• Quality Check: Higher rank ≠ better quality, test different ranks

Memory Optimization

• LoRAs add <100MB to VRAM usage typically
• Unload unused LoRAs with pipe.unload_lora_weights()
• Use FP8 base models with FP8 LoRAs for maximum efficiency
• Limit simultaneous LoRAs to 3-4 for stability

Generation Optimization

• Keep num_inference_steps at 25-35 for quality/speed balance
• Use guidance_scale 7-9 for SDXL (higher than SD1.5)
• Enable xformers or torch 2.0 attention for speed boost
• Consider using SDXL Turbo base for faster iteration

Workflow Best Practices

• Organize LoRAs by category (style, character, quality, concept)
• Document effective weight combinations
• Test LoRAs individually before stacking
• Keep notes on prompt keywords that work well with each LoRA
• Use version control for LoRA collections

License Information

LoRA License: Most SDXL LoRAs inherit the base model license

SDXL Base License: CreativeML Open RAIL++-M License

• ✅ Commercial use allowed
• ✅ Distribution and modification permitted
• ✅ Private and public use
• ⚠️ Must include license and copyright notice
• ⚠️ Cannot use for illegal purposes or harassment
• ⚠️ See full license terms: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md

Individual LoRA Licenses: Check each LoRA's source repository for specific terms. Some may have additional restrictions or different licenses.

Citation

If you use SDXL LoRAs in your work, please cite the original SDXL paper:

@article{podell2023sdxl,
  title={SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis},
  author={Podell, Dustin and English, Zion and Lacey, Kyle and Blattmann, Andreas and Dockhorn, Tim and Müller, Jonas and Penna, Joe and Rombach, Robin},
  journal={arXiv preprint arXiv:2307.01952},
  year={2023}
}

For specific LoRAs, also cite the original LoRA creators/trainers when applicable.

LoRA Training Resources

Official Documentation

• Diffusers Training: https://huggingface.co/docs/diffusers/training/lora
• SDXL LoRA Training Guide: https://huggingface.co/docs/diffusers/training/sdxl
• Kohya Training Scripts: https://github.com/kohya-ss/sd-scripts

Community Resources

• CivitAI: Large community LoRA repository and training guides
• Hugging Face Hub: Official LoRA collections and documentation
• Reddit r/StableDiffusion: Community discussions and tips

Recommended Training Tools

• kohya_ss GUI: User-friendly LoRA training interface
• OneTrainer: Modern training UI with SDXL support
• Diffusers Training Scripts: Official Hugging Face training code

Troubleshooting

Common Issues

LoRA Not Applying:

• Verify LoRA is compatible with SDXL (not SD1.5 LoRA)
• Check weight value is not 0
• Ensure proper file path and filename
• Verify SafeTensors file is not corrupted

Memory Errors:

• Reduce number of stacked LoRAs
• Lower resolution (1024x1024 → 768x768)
• Use FP8 base model for lower VRAM usage
• Enable CPU offloading with enable_sequential_cpu_offload()

Quality Issues:

• Adjust LoRA weight (try 0.5-0.9 range)
• Check LoRA compatibility with base model version
• Increase inference steps (30-50)
• Try LoRA individually to isolate conflicts

Slow Generation:

• LoRAs should add minimal overhead
• Check base model optimization (xformers, torch 2.0)
• Verify GPU is being used (not CPU fallback)
• Reduce number of stacked LoRAs

Contact and Resources

Official SDXL Resources

• Stability AI: https://stability.ai/
• SDXL Model Card: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0
• Diffusers Documentation: https://huggingface.co/docs/diffusers/

LoRA Communities

• CivitAI: https://civitai.com/ (largest LoRA repository)
• Hugging Face: https://huggingface.co/models?pipeline_tag=text-to-image&library=diffusers
• r/StableDiffusion: https://reddit.com/r/StableDiffusion/

Support

For issues with specific LoRAs, contact the original LoRA creator/trainer. For SDXL base model issues, refer to Stability AI's official channels.

---

Repository Status: Ready for LoRA collection (currently empty) Last Updated: 2025-10-28 Repository Size: ~14KB (directory structure only) Maintained By: Local model collection for SDXL LoRA adapters