fuse-dit

Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis

<div align="center"> <img src="https://github.com/tang-bd/fuse-dit/blob/main/assets/visual.jpg?raw=true" width="95%"/> </div>

Resources

• arXiv: Paper
• GitHub: Code

Quick Start

You can download the pre-trained model and then use FuseDiTPipeline in our codebase to run inference:

import torch
from diffusion.pipelines import FuseDiTPipeline
pipeline = FuseDiTPipeline.from_pretrained("/path/to/pipeline/").to("cuda")
image = pipeline(
    "your prompt",
    width=512,
    height=512,
    num_inference_steps=25,
    guidance_scale=6.0,
    use_cache=True,
)[0][0]
image.save("test.png")

Citation

@article{tang2025exploringdeepfusion,
    title={Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis}, 
    author={Bingda Tang and Boyang Zheng and Xichen Pan and Sayak Paul and Saining Xie},
    year={2025},
    journal={arXiv preprint arXiv:2505.10046}, 
}