ideogram4_caption_blocks

Ideogram4 caption block

A custom Modular Diffusers block that produces Ideogram 4's native structured-JSON caption from either an image or a short text idea, using a Gemma-4 vision-language model. The caption it returns can be fed straight into the Ideogram 4 generation pipeline as prompt.

image  -> caption it -> { high_level_description, style_description, compositional_deconstruction }
prompt -> enhance -> { high_level_description, compositional_deconstruction }

The mode is chosen automatically: pass image to caption it, or prompt (with no image) to enhance a short idea. Prompt enhancement reuses Ideogram's canonical magic-prompt system message from diffusers.pipelines.ideogram4.prompt_enhancer.

Loading & running

import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image

pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_caption_blocks", trust_remote_code=True)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("your_image.png")
caption = pipe(image=image, output="caption")   # caption the image -> JSON string
print(caption)

Enhance a text idea instead (no image) — Ideogram's "magic prompt":

caption = pipe(
    prompt="a cozy coffee shop on a rainy evening",
    output="caption",
)

Get the parsed dict instead (or alongside):

out = pipe(image=image, output=["caption", "caption_json"])
out["caption_json"]   # dict, or None if the model output couldn't be parsed

Inputs: image (caption it) or prompt (enhance it); instruction (image-mode schema prompt), height/width (aspect-ratio hint for enhance mode), max_new_tokens (2048), temperature (0.0 = greedy). Outputs: caption (pretty JSON string), caption_json (parsed dict), caption_raw (raw decoded text).

Caption → generate

caption = pipe(image=ref, output="caption")
# feed straight into the Ideogram 4 generation pipeline (see OzzyGT/ideogram4-modular)
image = gen_pipe(prompt=caption, output="images")[0]

Notes

• Default checkpoint: google/gemma-4-E4B-it (official bf16, ~15 GB — gated, needs a license-accepted HF token and a >=24 GB GPU). Requires transformers>=5.12. To run on a smaller GPU, point pretrained_model_name_or_path at a quantized checkpoint of the same model — import its quantization backend first, as the block no longer bundles one.
• Ideogram 4 was trained on these JSON captions, so a caption from this block is the ideal prompt for re-generation / auto-captioned img2img.