tarot-qlora-flux

LoRA for FLUX.1-dev - Tarot Card Style

This repository contains a LoRA (Low-Rank Adaptation) fine-tuned on black-forest-labs/FLUX.1-dev to generate images in the artistic style of tarot cards. This work is part of the blog post, "Fine-Tuning FLUX.1-dev on consumer hardware and in FP8".

<Gallery />

Inference

There are two main ways to use this LoRA for inference: loading the adapter on the fly or merging it with the base model.

Option 1: Loading LoRA Adapters

This approach offers flexibility, allowing you to easily switch between different LoRA styles.

from diffusers import FluxPipeline
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.enable_model_cpu_offload()

image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_loaded.png")

Option 2: Merging LoRA into Base Model

Merging the LoRA into the base model can lead to slightly faster inference and is useful when you want to use a single style consistently.

from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel
import torch

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

# You can save the fused transformer for later use
# pipeline.transformer.save_pretrained("fused_transformer")

pipeline.enable_model_cpu_offload()
image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_merged.png")

you can also requantize model:

from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel, BitsAndBytesConfig
import torch 

ckpt_id = "black-forest-labs/FLUX.1-dev"
pipeline = FluxPipeline.from_pretrained(
    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
)
pipeline.load_lora_weights("derekl35/tarot-qlora-flux", weight_name="pytorch_lora_weights.safetensors")
pipeline.fuse_lora()
pipeline.unload_lora_weights()

pipeline.transformer.save_pretrained("fused_transformer")

ckpt_id = "black-forest-labs/FLUX.1-dev"
bnb_4bit_compute_dtype = torch.bfloat16

nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=bnb_4bit_compute_dtype,
)
transformer = FluxTransformer2DModel.from_pretrained(
    "fused_transformer",
    quantization_config=nf4_config,
    torch_dtype=bnb_4bit_compute_dtype,
)

pipeline = AutoPipelineForText2Image.from_pretrained(
    ckpt_id, transformer=transformer, torch_dtype=bnb_4bit_compute_dtype
)
pipeline.enable_model_cpu_offload()

image = pipeline(
    'a trtcrd of a strong figure in work clothes kneeling beside a half-built stone tower, hammer in hand, with blueprints scattered around, a phoenix rising from forge flames in the background, mountains silhouetted against dawn, "the builder"',
    num_inference_steps=28,
    guidance_scale=3.5,
    height=768,
    width=512,
    generator=torch.manual_seed(0)
).images[0]
image.save("tarot_merged.png")