refcontrol-FLUX.2-klein-9B-reference-normal-lora

RefControl FLUX.2 Klein 9B – Reference Normal LoRA

📝 Short description

A LoRA for FLUX.2 Klein 9B Base that fuses a reference image (identity) with a normal map (structure / surface orientation).
It preserves identity and style from the reference while following the 3D shape and composition from the normal control map.

Trigger word: refcontrol

<Gallery />

---

📊 Examples

Each preview is a single combined image from ComfyUI: Normal → Reference → Result (left to right).

<table> <thead> <tr> <th>Normal → Reference → Result</th> </tr> </thead> <tbody> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora/resolve/main/images/a (1).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora/resolve/main/images/a (2).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora/resolve/main/images/a (3).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora/resolve/main/images/a (4).png" width="960" /></td> </tr> <tr> <td><img src="https://huggingface.co/thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora/resolve/main/images/a (5).png" width="960" /></td> </tr> </tbody> </table> ---

📖 Extended description

This LoRA was primarily trained on humans, but it also works with stylized characters and some objects.
Its main purpose is to preserve identity — facial features, hairstyle, clothing, or object details — from the reference image, while adapting the subject to the structure and surface orientation defined by the normal map.

FLUX.2 Klein 9B Base already handles reference + normal-guided transfer reasonably well with the right prompt alone. This LoRA builds on that capability — it improves consistency and better preserves character identity and normal-map fidelity than the base model without LoRA.

Part of the RefControl family: reference + control fusion for consistent, controllable generation on FLUX.2 Klein 9B Base.

---

⚙️ How to use

1. Use the first image as the normal map (structure / surface orientation).
2. Use the second image as the reference (character, person, or object).
3. Add the trigger word refcontrol in your prompt.
4. Adjust LoRA weight (recommended 0.8–1.0) depending on how strongly you want to preserve identity.

ComfyUI requirements

Normal map extraction in the included workflow uses DSINE-NormalMapPreprocessor (via comfyui_controlnet_aux):

https://github.com/Fannovel16/comfyui_controlnet_aux

You can disable the built-in normal preprocessor if your input is already a normal map image.

Base model

Trained on and recommended with black-forest-labs/FLUX.2-klein-base-9B.
The undistilled Base variant is intended for LoRA training and custom pipelines (~50 inference steps, guidance_scale ~4.0).

The LoRA also works with the 4-step distilled black-forest-labs/FLUX.2-klein-9B for faster inference (~4 steps, guidance_scale ~1.0), but quality may be slightly lower — especially for identity and normal-map fidelity — compared to the Base model.

✅ Example prompt

refcontrol

---

🎯 What it does

• Preserves character or object identity across generations.
• Adapts the subject to a new structure or composition defined by the normal map.
• Works best when the normal map has similar proportions and scale to the reference.

---

⚡ Tips

• Best results when the normal map is not drastically different in body scale or framing from the reference.
• Combine with text prompts to refine background, lighting, or mood.
• Normal maps with clear surface orientation and consistent lighting work well as control input.

---

📌 Use cases

• Character restyling while keeping identity and normal-defined structure.
• Consistent character design across different compositions.
• Illustration and storyboard generation with normal-guided layout.
• Object transformations with surface-orientation-guided placement.

---

📦 Files

• Weights: flux2_klein_9b_refcontrol_normal.safetensors
• ComfyUI workflow: refcontrol_normal_flux_klein_9b.json
• Repo: thedeoxen/refcontrol-FLUX.2-klein-9B-reference-normal-lora