stable-diffusion-3.5-large

See our collection for all Stable Diffusion 3.5 checkpoints.

Run Stable Diffusion 3.5 with Keras 3: JAX, PyTorch, or TensorFlow

Powered by Stability AI

zeromodels/stable-diffusion-3.5-large

Paper: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (arXiv:2403.03206) | HF Papers

Pure-Keras 3 conversion of stabilityai/stable-diffusion-3.5-large for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX. The container ships the MMDiT denoiser, the 16-channel VAE and the two CLIP text encoders in model.weights.json shards (9.05B parameters, 17.01 GB), plus zm_config.json (the component configs, the checkpoint's FlowMatchEulerDiscreteScheduler (shift 3.0) and the default generation settings) and the two tokenizers (tokenizer.json, the CLIP BPE, and tokenizer_3.json, the T5 SentencePiece). Weights are stored in float16, the checkpoint's native precision (the VAE in float32), and load in float16 by default; pass load_dtype="float32" to from_weights for a float32 model. This checkpoint generates 1024x1024 images (a 128x128x16 latent).

The third text encoder, the 4.7B-parameter T5-XXL, is shared by every SD 3 / 3.5 checkpoint and hosted once at zeromodels/t5-v1_1-xxl-encoder; attach it with text_encoder_3= (below) or leave it out (the T5 features are zeroed, SD 3's memory-saving mode).

For model details, intended use and limitations, see the upstream model card.

Architecture

Quick start

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
from zeromodels.models.stable_diffusion_3_5 import StableDiffusion3_5TextToImage, StableDiffusion3_5Tokenizer

model = StableDiffusion3_5TextToImage.from_weights(
    "zeromodels/stable-diffusion-3.5-large",
    text_encoder_3="zeromodels/t5-v1_1-xxl-encoder",  # optional: omit to zero the T5 features
)
tokenizer = StableDiffusion3_5Tokenizer.from_weights("zeromodels/stable-diffusion-3.5-large")

inputs = tokenizer("a photograph of an astronaut riding a horse")
images = model.generate(**inputs, num_inference_steps=28, guidance_scale=3.5, seed=0)
Image.fromarray(images[0]).save("astronaut.png")  # (1024, 1024, 3) uint8

generate takes the tokenizer's input_ids / attention_mask / input_ids_3 (batch them for several prompts), an optional tokenized negative prompt (negative_input_ids / negative_input_ids_3), num_inference_steps, guidance_scale, a seed, or explicit latents of shape (batch, 128, 128, 16) for results that are identical across backends; image / strength refine an image instead.

Load any Stable Diffusion 3.5 checkpoint the same way with from_weights("zeromodels/<variant>"):

Tips

• Set KERAS_BACKEND before importing Keras / zeromodels.
• The graphs are built for 1024px. Pass transformer_sample_size=<px / 8>, vae_sample_size=<px> to from_weights to build for another multiple of 16px (the weights are resolution-independent up to the 3072px position grid).
• StableDiffusion3_5Model.from_weights(...) loads the same repo as the bare container (.transformer / .vae / .text_encoder / .text_encoder_2) without the generation loop.
• model.text_encoder_3 can be any SD3T5EncoderModel (for example one loaded with quantization="int8"); it is not part of the container's weights.
• Both channels_last and channels_first are supported (keras.config.set_image_data_format before loading); generate always returns (batch, H, W, 3) uint8.
• On-the-fly hf: conversion is not supported for diffusion models; the checkpoints are hosted here, converted once.
• See the Stable Diffusion 3.5 docs.

License

The weights are redistributed under the Stability AI Community License of the upstream checkpoint, including its use-based restrictions. By using them you agree to those terms.

Notice

This Stability AI Model is licensed under the Stability AI Community License, Copyright © Stability AI Ltd. All Rights Reserved

Modifications by zeromodels (https://github.com/IMvision12/ZeroModels): the checkpoint released at https://huggingface.co/stabilityai/stable-diffusion-3.5-large was converted to the Keras 3 weights layout of zeromodels (model.weights.json, model_00000.weights.h5, model_00001.weights.h5, model_00002.weights.h5, model_00003.weights.h5, zm_config.json, tokenizer.json, tokenizer_3.json), stored in float16, the upstream fp16 files, with the VAE in float32. The model architecture and the parameter values are unchanged; the weight names and the file format differ from the release.

Powered by Stability AI

Special Thanks

Thank you to Stability AI for training and releasing Stable Diffusion, and to the Hugging Face diffusers team, whose implementation this port was verified against.