sd15-wikiart-impressionism-lora

Sd15 Wikiart Impressionism Lora

A LoRA adapter for Stable Diffusion 1.5, fine-tuned on 5,000 Impressionism paintings from WikiArt.

Usage

import torch
from diffusers import StableDiffusionPipeline
from peft import PeftModel

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    safety_checker=None,
)

# Merge the adapter at 0.6 strength (see "Adapter strength" below)
peft_unet = PeftModel.from_pretrained(pipe.unet, "MayankTamakuwala/sd15-wikiart-impressionism-lora")
for module in peft_unet.modules():
    if hasattr(module, "scaling") and isinstance(module.scaling, dict):
        for name in module.scaling:
            module.scaling[name] *= 0.6
pipe.unet = peft_unet.merge_and_unload()
pipe = pipe.to("cuda")

image = pipe(
    "an Impressionism painting, landscape, by Claude Monet",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]
image.save("output.png")

Adapter strength

Use ~0.6, not 1.0. At full strength this adapter imposes palette and brushwork hard enough to dissolve composition in complex multi-figure scenes. A matched-seed sweep over 0.4 / 0.6 / 0.8 / 1.0 put the breakdown between 0.6 and 0.8, while the learned behaviours survive down to 0.4.

Prompting

Trained on captions synthesized from WikiArt metadata in the form:

an Impressionism painting, {genre}, by {artist}

Prompts in that shape work best, e.g. an Impressionism painting, landscape, by Claude Monet. Free-form prompts also work; the style transfers to novel compositions.

What the adapter learned

Two effects are clearly attributable to fine-tuning, since neither is prompted and neither appears in base SD 1.5 output at matched seeds:

• No picture frames. Base SD renders "a painting" as a photograph of a framed painting on a wall. WikiArt images are cropped to the canvas, so the adapter produces edge-to-edge artwork.
• Photographed-painting colour. Output shifts toward the muted, slightly aged palette of real scanned paintings, away from SD's idealised saturation.

Evaluation

1,000 generated images against 653 held-out WikiArt Impressionism images the model never saw (seeded train/val split):

FID is computed with clean-fid; CLIP score with OpenCLIP ViT-B-32. Note the absolute FID is inflated by the small reference set (653 images, below the ~1000 where FID stabilises) -- the delta against base is the meaningful quantity, not the absolute value.

Training

Source

Training, evaluation, and serving code: https://github.com/MayankTamakuwala/Diffusion-Based-Image-Generation-Editing

Limitations

• Complex multi-figure scenes degrade at high adapter strength.
• Trained on one style; other WikiArt styles need their own adapter.
• Inherits Stable Diffusion 1.5's limitations and biases.
• Non-commercial research use, per the CreativeML OpenRAIL-M licence.