flickr30k-stable-diffusion-fine-tuned

Fine-Tuned Stable Diffusion (UNet) — Flickr30k 🚀

Model type: Text-to-Image diffusion (Stable Diffusion v1.4 UNet fine-tune)
Base model: CompVis/stable-diffusion-v1-4
Dataset: lmms-lab/flickr30k
Framework: PyTorch (diffusers)
Resolution: 512×512
Trained epochs: 3
Batch size: 1
Learning rate: 2e-5
Seed: 42
Optimizer: torch.optim.AdamW
Last updated: 2025-08-16

⚠️ What is in this repo?
This repository publishes a fine-tuned UNet for Stable Diffusion v1-4 rather than a full pipeline.
To generate images, load the base pipeline (CompVis/stable-diffusion-v1-4) and replace the UNet with the weights from this repo.

---

🔧 How to use

import torch
from diffusers import StableDiffusionPipeline, UNet2DConditionModel

repo_id = "YOUR_USERNAME/YOUR_REPO_NAME"  # <— replace with your HF repo id

# 1) Load the base pipeline
pipe = StableDiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", torch_dtype=torch.float16
).to("cuda")

# 2) Load the fine-tuned UNet from this repo and swap it in
ft_unet = UNet2DConditionModel.from_pretrained(repo_id)
pipe.unet = ft_unet

# 3) Inference
prompt = "a high-resolution photo of a golden retriever surfing a wave, cinematic lighting"
image = pipe(prompt, num_inference_steps=30, guidance_scale=7.5).images[0]
image.save("sample.png")

Alternative: Local weights

If you downloaded the UNet folder locally:

ft_unet = UNet2DConditionModel.from_pretrained("./ultra_fine_tuned_unet")
pipe.unet = ft_unet

---

📊 Training summary

• Base: CompVis/stable-diffusion-v1-4 (CreativeML Open RAIL-M)
• Dataset: lmms-lab/flickr30k
• Objective: Fine-tune UNet for improved text–image alignment/quality on Flickr30k-style prompts
• Hyperparameters:
  • epochs: 3
  • learning rate: 2e-5
  • batch size: 1
  • resolution: 512
  • optimizer: torch.optim.AdamW
  • seed: 42

Note: The notebook saved only the UNet via .save_pretrained("./ultra_fine_tuned_unet"). If you also want to publish the VAE and text encoder, re-save those components or export a full pipeline.

---

✅ Intended Use & Limitations

Intended use

• Research & experimentation in text-to-image generation
• Educational demos of diffusion fine-tuning workflows

Not intended for

• Safety-critical or high-risk applications
• Generation of harmful, illegal, or discriminatory content
• Any use that violates the license of the base model or the training data

Known limitations

• May retain biases present in Flickr30k captions/images (cultural, gender, age, etc.)
• Performance is tuned for 512×512; other resolutions may degrade quality
• Only UNet is fine-tuned; text encoder/CLIP alignment is unchanged

---

⚖️ Ethics & Responsible AI

• Bias & Fairness: Flickr30k includes real-world images/captions that can encode stereotypes and imbalances. Outputs may reflect these patterns. Evaluate downstream uses for disparate impact.
• Consent & Copyright: Flickr30k images are copyrighted by original photographers and released for research. Do not imply that outputs are endorsed by subjects; avoid identity-sensitive use cases.
• Misuse: Prohibit generation of hateful, harassing, or sexual content; violence; illegal activities. Do not use for surveillance or facial recognition.
• Transparency: Disclose that images are machine-generated and may be inaccurate or fabricated.
• Human Oversight: Keep a human in the loop when deploying to end-users; provide reporting/opt-out channels.

Safety recommendations

• Use negative prompts and safety checkers (safety_checker=None is discouraged in production).
• Apply content filters/watermarking where applicable.
• Log prompts for auditability (with user consent).

---

📦 Files in this repository

• pytorch_model.bin — UNet weights (uploaded by you)
• config.json — UNet architecture config (from UNet2DConditionModel)
• scheduler_config.json — (optional) if you choose to publish a recommended scheduler config
• README.md — This model card
• LICENSE — License (see below)

If you trained locally and only have ./ultra_fine_tuned_unet/, upload its contents directly to this repo root.

---

🔁 Reproducibility snippet (training skeleton)

from diffusers import StableDiffusionPipeline, UNet2DConditionModel, DDPMScheduler, AutoencoderKL
import torch

base = "CompVis/stable-diffusion-v1-4"
pipe = StableDiffusionPipeline.from_pretrained(base, torch_dtype=torch.float16)
unet = pipe.unet  # start from base UNet
# ... set up dataset, optimizer (AdamW), LR (2e-5), epochs (3), etc.
# train loop updating UNet weights only
unet.save_pretrained("./ultra_fine_tuned_unet")  # published in this repo

---

🧾 License

• Base model: CompVis/stable-diffusion-v1-4 — CreativeML Open RAIL-M.
• This fine-tuned UNet: Released under CreativeML Open RAIL-M to maintain compatibility with the base model’s responsible use constraints.

See LICENSE for details.

---

📥 How to publish to Hugging Face

pip install -U "huggingface_hub[cli]" diffusers transformers accelerate torch
huggingface-cli login  # paste your HF token

from huggingface_hub import HfApi, HfFolder, create_repo, upload_folder

repo_id = "YOUR_USERNAME/YOUR_REPO_NAME"
create_repo(repo_id, repo_type="model", private=False, exist_ok=True)

# Upload the UNet folder (generated by save_pretrained)
upload_folder(
    repo_id=repo_id,
    folder_path="./ultra_fine_tuned_unet",
    path_in_repo=".",
    repo_type="model"
)

# Also upload this README and LICENSE if running locally:
# api = HfApi()
# api.upload_file(repo_id=repo_id, path_or_fileobj="README.md", path_in_repo="README.md")
# api.upload_file(repo_id=repo_id, path_or_fileobj="LICENSE", path_in_repo="LICENSE")