flux-control-lora-brightness-10k

FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)

A Control LoRA model trained on FLUX.1-dev to control image generation through brightness/grayscale information. This model uses LoRA (Low-Rank Adaptation) for efficient control, providing an ultra-lightweight alternative to ControlNet with excellent pattern preservation and minimal overhead.

Model Description

This Control LoRA enables brightness-based conditioning for FLUX.1 image generation. By providing a grayscale image as input, you can control the brightness distribution and lighting structure while maintaining creative freedom through text prompts.

Key Features:

• 🎨 Excellent brightness and pattern control across all scales (0.25-1.5)
• 🚀 188x smaller than SDXL ControlNet: ~25MB vs ~4.7GB
• ⚡ Ultra-fast loading: LoRA weights load in <1 second
• 💡 Flexible scale control: Adjustable adapter weights from 0.25 to 1.5+
• 🔄 Compatible with any FLUX.1-dev setup: Just load the LoRA weights
• 📦 Minimal storage: All checkpoints + final model = ~125MB total

Intended Uses:

• Artistic QR code generation (scale 0.7-1.0 recommended)
• Image recoloring and colorization
• Lighting control in text-to-image generation
• Brightness-based pattern integration
• Watermark and subtle pattern embedding
• Photo enhancement and stylization

Training Details

Training Data

Trained on 10,000 samples from latentcat/grayscale_image_aesthetic_3M:

• High-quality aesthetic images
• Paired with grayscale/brightness versions
• Resolution: 512×512 (balanced for FLUX.1)

Training Configuration

Model Size Comparison

Usage

Installation

pip install diffusers transformers accelerate torch sentencepiece peft

Basic Usage

from diffusers import FluxControlPipeline
import torch
from PIL import Image

# Load FLUX.1 Control pipeline
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set adapter weight (scale) - controls conditioning strength
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((512, 512))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    control_image=control_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("output.png")

Adjusting Control Strength

The adapter weight controls how strongly the brightness map influences generation:

# Subtle control (scale 0.25-0.5)
pipe.set_adapters(["default_0"], adapter_weights=[0.25])

# Balanced control (scale 0.7-1.0) - Recommended for artistic QR codes
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Strong control (scale 1.0-1.5)
pipe.set_adapters(["default_0"], adapter_weights=[1.25])

Artistic QR Code Generation

import qrcode
from PIL import Image

# Generate QR code
qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_H,
    box_size=10,
    border=4
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((512, 512), Image.LANCZOS).convert("RGB")

# Load pipeline and LoRA
pipe = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")
pipe.to("cuda")

# Set scale for good QR code readability (0.7-1.0 works best)
pipe.set_adapters(["default_0"], adapter_weights=[0.75])

# Generate artistic QR code
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    control_image=qr_image,
    num_inference_steps=28,
    guidance_scale=3.5,
    height=512,
    width=512,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 2,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-625")

# Mid checkpoint (50% - 5,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1250")

# Late checkpoint (75% - 7,500 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-1875")

# Final checkpoint (100% - 10,000 samples)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k", revision="checkpoint-2500")

# Final model (same as checkpoint-2500, on main branch)
pipe.load_lora_weights("Oysiyl/flux-control-lora-brightness-10k")

Adapter Weight (Scale) Guide

The adapter weight controls how strongly the brightness map influences generation:

Recommended Scale Ranges

Visual Examples

Checkpoint Progression at Scale 0.75

This image shows how the model improves across checkpoints at scale 0.75 (recommended for QR codes):

From left to right: Original QR input, checkpoint-625 (25%), checkpoint-1250 (50%), checkpoint-1875 (75%), checkpoint-2500 (100%), final model

Final Model - All Scales Comparison

This grid shows the final model's behavior across all scales (0.25 to 1.5):

Grid showing scales: 0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5 - demonstrating the range from subtle hints to strong pattern preservation

Input QR Code:

Performance Comparison

vs ControlNet (SDXL)

vs T2I Adapter (SDXL)

Checkpoint Progression Analysis

The model includes checkpoints from throughout training:

1. checkpoint-625: 25% complete (2,500 samples)
2. checkpoint-1250: 50% complete (5,000 samples)
3. checkpoint-1875: 75% complete (7,500 samples)
4. checkpoint-2500: 100% complete (10,000 samples)
5. Final model: Same as checkpoint-2500 (main branch)

Key Observations

All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:

1. Early Checkpoint (625 steps, 2.5k samples):

   • Good pattern awareness, developing control
   • More artistic interpretation of prompts
   • Recommended scales: 0.5-1.0

2. Mid Checkpoints (1250-1875 steps, 5k-7.5k samples):

   • Strong balance between control and creativity
   • Stable pattern preservation
   • Recommended scales: 0.7-1.25

3. Final Model (2500 steps, 10k samples):

   • Maximum control capability
   • Excellent pattern preservation at all scales
   • Recommended scales: 0.7-1.5

No Overfitting Observed

Unlike larger models trained on 100k samples, this 10k Control LoRA shows no signs of overfitting:

• Consistent improvement throughout training
• Final checkpoint is recommended for production use
• 10k samples appears optimal for LoRA-based control training

When to Use This Model

✅ Use This Control LoRA When:

• Creating artistic QR codes with FLUX.1 quality (scale 0.7-1.0)
• Need minimal storage overhead (<30MB per checkpoint)
• Want fast model switching between checkpoints (<1 second)
• Building production applications requiring small model sizes
• Working with FLUX.1-dev as base model
• Require flexible control strength via adapter weights
• Need multiple checkpoints without massive storage (125MB total vs 18.8GB)

⚠️ Consider Alternatives When:

• Need higher resolution (>512×512) - consider SDXL T2I Adapter @ 1024×1024
• Require extremely precise control at very low scales (<0.25)
• Working with existing SDXL pipelines (use T2I Adapter or ControlNet)

Limitations

Current Limitations

• Resolution: Trained at 512×512, may not perform optimally at higher resolutions
• Base model dependency: Requires FLUX.1-dev base model
• Grayscale conditioning only: Trained specifically for brightness/grayscale control
• Single control type: Only brightness, not other conditioning types (pose, depth, etc.)

Recommendations

• For 512×512 generation, use this Control LoRA
• For 1024×1024+ generation, consider the SDXL T2I Adapter models
• Experiment with scales 0.7-1.0 for most use cases
• Use earlier checkpoints (625-1250) if you prefer more artistic freedom

Training Script

accelerate launch --mixed_precision="bf16" train_control_lora_flux.py \
  --pretrained_model_name_or_path="black-forest-labs/FLUX.1-dev" \
  --dataset_name="<path_to_10k_dataset>" \
  --conditioning_image_column="conditioning_image" \
  --image_column="image" \
  --caption_column="text" \
  --output_dir="./flux-control-lora-brightness-10k" \
  --mixed_precision="bf16" \
  --resolution=512 \
  --learning_rate=1e-4 \
  --proportion_empty_prompts=0.5 \
  --rank=128 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --num_train_epochs=1 \
  --checkpointing_steps=625 \
  --validation_steps=625 \
  --gradient_checkpointing \
  --report_to="wandb"

Available Branches

• main: Final model (10,000 samples, recommended)
• checkpoint-625: Early checkpoint (2,500 samples)
• checkpoint-1250: Mid checkpoint (5,000 samples)
• checkpoint-1875: Late checkpoint (7,500 samples)
• checkpoint-2500: Final checkpoint (same as main)

Citation

@misc{flux-control-lora-brightness-10k,
  author = {Oysiyl},
  title = {FLUX.1 Control LoRA - Brightness Control (10k @ 512×512)},
  year = {2025},
  publisher = {HuggingFace},
  journal = {HuggingFace Model Hub},
  howpublished = {\url{https://huggingface.co/Oysiyl/flux-control-lora-brightness-10k}}
}

Acknowledgments

• Built with 🤗 Diffusers
• Base model: FLUX.1-dev by Black Forest Labs
• Dataset: grayscale_image_aesthetic_3M by latentcat
• Training infrastructure: NVIDIA H100 80GB
• LoRA implementation: PEFT by Hugging Face

License

Apache 2.0 License. The base FLUX.1-dev model has separate license terms at black-forest-labs/FLUX.1-dev.