sd2.1-base-Diorma-style-LoRA

SD 2.1 Diorama LoRA — Diorama Machine

A LoRA adapter fine-tuned on Stable Diffusion 2.1 to generate miniature, isometric diorama-style architectural interiors. Built as part of an independent deep learning project exploring diffusion model fine-tuning and evaluation methodology.

Live demo: Diorama Machine Space

---

Why LoRA

Full fine-tuning of SD 2.1's UNet (869M parameters) wasn't feasible on the available 8GB GPU. LoRA freezes the base model and trains small low-rank matrices (A, B) injected alongside the attention projection layers (to_q, to_k, to_v, to_out.0) — the layers where text conditioning meets image features:

W_effective = W + (alpha / r) * (A @ B)

This project used r=32, alpha=64, bringing trainable parameters down to ~6.6M — 0.76% of the full model — which is what made training on diorama-style interior renders practical on consumer hardware in the first place, without needing to touch or store gradients for the frozen 869M-parameter base.

---

Model Details

---

Implementation — Dataset to Result

1. Dataset collection ~2,000 diorama-style architectural interior renders were sourced for training — isometric, miniature-scale interior scenes (bedrooms, living rooms, kitchens, bathrooms).

2. Caption inspection and cleaning Raw captions contained a systematic corruption pattern: duplicated scene descriptions followed by scraped platform attribution text (e.g. "a living room with a red couch a living room with a red couch ... on Behance"), affecting the large majority of entries. A cleaning function detected the repeated-phrase pattern programmatically — comparing progressively shorter prefixes of each caption against the text immediately following them — and kept only a single, clean copy of the actual scene description, discarding the trailing attribution text.

3. Trigger word / caption structure Cleaned captions were used as-is, without an artificial style trigger word prepended — the model learns the diorama aesthetic directly from the paired image/caption signal rather than from a keyword shortcut. At inference, no special trigger word is required in the prompt; plain scene descriptions (e.g. "a bedroom with a bed and a mirror") are sufficient.

4. Target objective — v-prediction SD 2.1 is trained with a v-prediction objective rather than epsilon (noise) prediction, unlike SD 1.5. The training loop computes loss against noise_scheduler.get_velocity(latents, noise, timesteps), matching the base model's actual prediction target.

5. Training loop Implemented from scratch in PyTorch: VAE-encoded latents, per-sample random timestep sampling, v-prediction noise-velocity targets, LoRA-only backpropagation, gradient clipping (max_norm=1.0), 8-bit AdamW, and a cosine learning rate schedule with warmup, run for 12 epochs with checkpointing every 3 epochs.

6. Result The final checkpoint (epoch 12) was evaluated and deployed as the adapter published here.

---

Usage

from diffusers import StableDiffusionPipeline
from peft import PeftModel
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "sd2-community/stable-diffusion-2-1",
    torch_dtype=torch.float16,
).to("cuda")
pipe.unet = PeftModel.from_pretrained(pipe.unet, "Sathya77/sd21-diorama-lora")

img = pipe(
    "a room with a bed and a table",
    negative_prompt="blurry, distorted lines, melted architecture, sloppy, deformed, noise, messy details",
    num_inference_steps=50,
    guidance_scale=7.5,
).images[0]
img.save("output.png")