siwa-kershef-sdxl-lora

Siwa Kershef Architecture — SDXL LoRA (style-isolation experiment)

A LoRA for SDXL 1.0 that teaches the earthen kershef architecture of Siwa Oasis, Egypt, behind the trigger token k3rshef, evaluated specifically for style bleeding into other architectural styles.

Dataset (112 curated, hand-captioned images, per-image licence): 4Fever4/siwa-kershef-architecture

Usage

import torch
from diffusers import StableDiffusionXLPipeline
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("4Fever4/siwa-kershef-sdxl-lora", weight_name="siwa_kershef_sdxl_lora_v1.safetensors", adapter_name="siwa")
pipe.set_adapters(["siwa"], adapter_weights=[0.6])   # recommended strength, see evaluation
img = pipe("k3rshef, exterior of a two-storey house, projecting palm-trunk beams, small window openings, restored, photo",
           num_inference_steps=30, guidance_scale=6.5).images[0]

Prompt vocabulary the LoRA was captioned with (usable at inference): palm-trunk beams, thick tapered columns, small window openings, external staircase, crenellated parapet, tapered minaret, covered passage; condition: restored, partially restored, newly built, eroded ruin.

Training

Exact command: train.sh, dataset config: dataset.toml (kohya-ss/sd-scripts).

Evaluation — does the style stay behind its trigger?

scripts/evaluate.py generates the same prompts with the same seeds with base SDXL and with the LoRA loaded.

• Target prompts (with k3rshef) — the LoRA should pull these toward Siwa.
• Control prompts (no trigger, LoRA still loaded): Al-Qasr Dakhla (a neighbouring earthen style — the hardest case), Nubian Aswan, modern New Cairo, Mamluk Cairo, Scandinavian — the LoRA should not change these.

Metrics (CLIP ViT-L/14, cosine):

• siwa_shift = similarity to the training-set centroid (LoRA) − same (base). Target ↑, control ≈ 0.
• base_vs_lora_sim = similarity between the base image and the LoRA image for the same prompt+seed. Control ≈ 1 means untouched.

Summary

Per prompt (siwa_shift / base-vs-LoRA similarity)

Rows: base SDXL / v1 @1.0 / v1 @0.6. Columns: house + mosque (target, with trigger), Nubian, modern New Cairo, Mamluk (controls, no trigger). Same seed.

What happened — three iterations

1. v1 @1.0 learned the style strongly (target +0.058) but bled into neighbouring Egyptian prompts (control +0.044): modern New Cairo turned into a beige earthen block, Nubian lost its blue paint and composition, Mamluk domes and carving flattened into mud walls. Scandinavian and — notably — Al-Qasr (0.96 similarity) stayed stable, so the leak is semantic ("Egypt / desert → kershef"), not global.
2. v2 (prior-preservation): retrained with 48 base-SDXL regularisation images of neighbouring styles (Nubian, modern Cairo, Mamluk, Ottoman, Mediterranean, riad, desert resort …, captioned without the trigger, wording/seeds disjoint from the eval prompts). Bleeding dropped to +0.009, but the target style collapsed too (+0.007) — the regulariser was too strong for a 112-image set at 2,000 steps. Raising v2 to 1.5 made things worse. Negative result, kept for transparency.
3. v1 @0.6 — recommended. Keeps 84 % of the target shift (+0.049) while cutting control bleeding by 61 % (+0.044 → +0.017); controls stay 0.945 similar to base (Mamluk 0.964, Nordic 0.964, Al-Qasr 0.969). Remaining weak spot: modern New Cairo still drifts (+0.056).

What I would do next

• Weaker prior preservation (fewer reg repeats / prior_loss_weight 0.3–0.5) instead of 1:1, then re-sweep scale.
• Add negative-pair captions for the modern case (e.g. modern glass building reg images specifically).
• More intact-building data (the dataset is still 45 % ruins) and a human rating pass alongside CLIP.
• Port the same data / captions / evaluation to a Flux LoRA on a 24 GB GPU.

Limitations

• One-day experiment; 112 images, 3 seeds per prompt — the numbers are indicative, not a benchmark.
• SDXL, not Flux: Flux LoRA training does not fit an 8 GB GPU. The same data/captions/evaluation pipeline transfers directly to Flux (e.g. ai-toolkit) on a 24 GB GPU.
• CLIP similarity is a proxy; the grids in eval/ should be looked at, not only the numbers.