Difuzcam_model

Difuzcam Model

DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model for Generative AI Based Flat Camera Design

Erez Yosef, Raja Giryes (Tel Aviv University)

Published in Scientific Reports, Volume 15, Nature Publishing Group, 2025. paper

Model Description

ControlNet model for flat camera image reconstruction using diffusion-based generative AI. This model enables lensless imaging by reconstructing high-quality RGB images from raw flat camera sensor measurements.

Model type: ControlNet for Stable Diffusion 2.1 License: Apache-2.0

Model Details

• Base Model: stabilityai/stable-diffusion-2-1-base
• Architecture: FlatControlNet with learnable separable transformation
• Training Data: 54,170 simulated flat camera measurements from ErezYosef/Difuzcam_dataset 🤗.
• Resolution: 512×512

Usage

Clone the Difuzcam GitHub repository

from diffusers import UNet2DConditionModel
from models.flat_controlnet import FlatControlNetModel_Efull

unet = UNet2DConditionModel.from_pretrained(
    'stabilityai/stable-diffusion-2-1-base',
    subfolder="unet"
)
controlnet = FlatControlNetModel_Efull.from_pretrained(
    'ErezYosef/controlnet',
    low_cpu_mem_usage=False,
    unet=unet
)

For complete training and inference code, see the Difuzcam GitHub repository.

Dataset

Training dataset available at ErezYosef/Difuzcam_dataset

Citation

@article{yosef2025difuzcam,
  title={DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model for Generative AI Based Flat Camera Design},
  author={Yosef, Erez and Giryes, Raja},
  journal={Scientific Reports},
  volume={15},
  number={1},
  pages={43059},
  year={2025},
  publisher={Nature Publishing Group UK London}
}

Contact

Erez Yosef - [email protected]