refcontrol-FLUX.2-klein-4B-reference-depth-lora

RefControl FLUX.2 Klein 4B - Reference Depth LoRA

📝 Short description

A LoRA for FLUX.2 Klein 4B Base that fuses a reference image (left) with a depth map (right).
It preserves identity and style from the reference while following the pose and structure from the depth map.

Trigger word: refcontrol

<Gallery />

---

📊 Examples

<table> <thead> <tr> <th>Reference</th> <th>Depth</th> <th>Result</th> </tr> </thead> <tbody> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1ref.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1d.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/1res.png" width="320" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2ref.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2d.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/2res.png" width="320" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3ref.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3d.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/3res.png" width="320" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4ref.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4d.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/4res.png" width="320" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5ref.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5d.png" width="320" /></td> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora/resolve/main/images/5res.png" width="320" /></td> </tr> </tbody> </table>

---

📖 Extended description

This LoRA was primarily trained on humans, but it also works with objects and stylized characters.
Its main purpose is to preserve identity - facial features, clothing, style, or object characteristics - from the reference image, while adapting the subject to the pose and composition defined by the depth map.

Part of the RefControl family: reference + control fusion for consistent, controllable generation on FLUX.2 Klein 4B Base.

---

⚙️ How to use

1. Use first image as depth reference, second image input.
2. Add the trigger word refcontrol in your prompt.
3. Adjust LoRA weight (recommended 0.8-1.0) depending on how strongly you want to preserve identity.

ComfyUI requirenments

For extracting depth you need to install nodes with depth estimation in workflow used:

https://github.com/Fannovel16/comfyui_controlnet_aux

Base model

Use with black-forest-labs/FLUX.2-klein-base-4B.
The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, guidance_scale ~4.0).

✅ Example prompt

refcontrol

---

🎯 What it does

• Preserves character or object identity across generations.
• Embeds the subject into a new pose or scene defined by the depth map.
• Works best when the depth map has similar proportions and scale to the reference.

---

⚡ Tips

• Best results when the depth map is not drastically different in object scale from the reference.
• Combine with text prompts to refine background, lighting, or mood.
• Can be combined with other RefControl LoRAs (pose, lineart, canny) on compatible pipelines for multi-constraint generation.

---

📌 Use cases

• Human portraits in different poses.
• Consistent character design across multiple scenes.
• Object transformations (cars, furniture, props) with depth-guided placement.
• Storyboarding, comics, or animation frame generation.

---

📦 Files

• Weights: flux2_klein_4b_refcontrol_depth.safetensors
• ComfyUI workflow: image_flux2_klein_image_edit_4b_refcontrol_depth.json
• Repo: thedeoxen/refcontrol-FLUX.2-klein-4B-reference-depth-lora