Qwen-Image-Edit-2511-Scene2CharRef-LoRA

Qwen-Image-Edit-2511 LoRA - Scene -> Character Reference (reverse)

The reverse of CharSheet2Art: takes a finished illustrated scene and extracts its character(s) as a clean character reference - each character in a neutral standing pose on a plain light-gray studio background, eye-level, preserving identity, art style, outfit and orientation, with the scene/background removed.

Purpose: a moderation-free reference extractor for building synthetic reference->scene datasets without hitting hosted-model safety filters.

Prompt format (same as training): Using Image 1, create a clean character reference of the character(s) in it: ... neutral pose on a plain light-gray studio background ...

Usage

import torch
from PIL import Image
from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig

pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16, device="cuda",
    model_configs=[
        ModelConfig(model_id="Qwen/Qwen-Image-Edit-2511", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
    ],
    tokenizer_config=None,
    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
)
pipe.load_lora(pipe.dit, "checkpoints/epoch-4.safetensors")
scene = Image.open("scene.jpeg")
img = pipe("Using Image 1, create a clean character reference of the character in it: "
           "the character standing in a neutral, relaxed pose on a plain light-gray "
           "studio background at eye level, preserving their exact identity, art style, "
           "outfit and the same head and body orientation and framing. Remove the original scene and background.",
           edit_image=[scene], seed=0, num_inference_steps=40,
           height=1536, width=1024, zero_cond_t=True)  # zero_cond_t REQUIRED for 2511

Training config

Full args: training_config.json

Loss

Validation samples

8 held-out scenes x 5 checkpoints in val_samples/ - scenes not seen in training; each generates the extracted reference. Examples (epoch-4):

Dataset sample

One reverse pair in dataset_example/: input scene, target reference, and the prompt (pair.json).