refcontrol-FLUX.2-klein-9B-reference-canny-lora

RefControl FLUX.2 Klein 9B – Reference Canny LoRA

📝 Short description

A LoRA for FLUX.2 Klein 9B Base that fuses a reference image (identity) with a canny edge map (structure / contours).
It preserves identity and style from the reference while following the shape and composition from the canny control map.

Trigger word: refcontrol

<Gallery />

---

📊 Examples

Each preview is a single combined image from ComfyUI: Canny → Reference → Result (left to right).

<table> <thead> <tr> <th>Canny → Reference → Result</th> </tr> </thead> <tbody> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-canny-lora/resolve/main/images/a (1).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-canny-lora/resolve/main/images/a (2).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-canny-lora/resolve/main/images/a (3).png" width="960" /></td> </tr> </tbody> </table> ---

📖 Extended description

This LoRA was primarily trained on humans, but it also works with stylized characters and some objects.
Its main purpose is to preserve identity — facial features, hairstyle, clothing, or object details — from the reference image, while adapting the subject to the structure and contours defined by the canny edge map.

FLUX.2 Klein 9B Base already handles reference + edge-guided transfer reasonably well with the right prompt alone. This LoRA builds on that capability — it improves consistency and better preserves character identity and edge fidelity than the base model without LoRA.

Part of the RefControl family: reference + control fusion for consistent, controllable generation on FLUX.2 Klein 9B Base.

---

⚙️ How to use

1. Use the first image as the canny edge map (structure / contours).
2. Use the second image as the reference (character, person, or object).
3. Add the trigger word refcontrol in your prompt.
4. Adjust LoRA weight (recommended 0.8–1.0) depending on how strongly you want to preserve identity.

ComfyUI requirenments

Canny extraction in the included workflow uses CannyEdgePreprocessor (via comfyui_controlnet_aux):

https://github.com/Fannovel16/comfyui_controlnet_aux

You can disable the built-in canny preprocessor if your input is already a canny/edge image.

Base model

Trained on and recommended with black-forest-labs/FLUX.2-klein-base-9B.
The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, guidance_scale ~4.0).

The LoRA also works with the 4-step distilled black-forest-labs/FLUX.2-klein-9B for faster inference (~4 steps, guidance_scale ~1.0), but quality may be slightly lower — especially for identity and edge fidelity — compared to the Base model.

✅ Example prompt

refcontrol

---

🎯 What it does

• Preserves character or object identity across generations.
• Adapts the subject to a new structure or composition defined by the canny map.
• Works best when the edge map has similar proportions and scale to the reference.

---

⚡ Tips

• Best results when the canny map is not drastically different in body scale or framing from the reference.
• Combine with text prompts to refine background, lighting, or mood.
• Canny edge maps on a black background work well as control input.

---

📌 Use cases

• Character restyling while keeping identity and edge-defined structure.
• Consistent character design across different compositions.
• Illustration and storyboard generation with edge-guided layout.
• Object transformations with contour-guided placement.

---

📦 Files

• Weights: flux2_klein_9b_refcontrol_canny.safetensors
• ComfyUI workflow: refcontrol_canny_flux_klein_9b.json
• Repo: thedeoxen/refcontrol-FLUX.2-klein-9B-reference-canny-lora