controlnet-encoded-hands-130k

Model Description

As Stable diffusion and other diffusion models are notoriously poor at generating realistic hands for our project we decided to train a ControlNet model using MediaPipes landmarks in order to generate more realistic hands avoiding common issues such as unrealistic positions and irregular digits.

We opted to use the HAnd Gesture Recognition Image Dataset and MediaPipe's Hand Landmarker to train a control net that could potentially be used independently or as an in-painting tool.

Preprocess

To preprocess the data there were three options we considered:

• The first was to use Mediapipes built-in draw landmarks function. This was an obvious first choice however we noticed with low training steps that the model couldn't easily distinguish handedness and would often generate the wrong hand for the conditioning image.

<figure> <img src="https://datasets-server.huggingface.co/assets/MakiPan/hagrid250k-blip2/--/MakiPan--hagrid250k-blip2/train/29/image/image.jpg" alt="Forwarding"> <figcaption>Original Image </figcaption> </figure> <figure> <img src="https://datasets-server.huggingface.co/assets/MakiPan/hagrid250k-blip2/--/MakiPan--hagrid250k-blip2/train/29/conditioning_image/image.jpg" alt="Routing"> <figcaption>Conditioning Image </figcaption> </figure>

• To counter this issue we changed the palm landmark colors with the intention to keep the color similar in order to learn that they provide similar information, but different to make the model know which hands were left or right.

<figure> <img src="https://datasets-server.huggingface.co/assets/MakiPan/hagrid-hand-enc-250k/--/MakiPan--hagrid-hand-enc-250k/train/96/image/image.jpg" alt="Forwarding"> <figcaption>Original Image </figcaption> </figure> <figure> <img src="https://datasets-server.huggingface.co/assets/MakiPan/hagrid-hand-enc-250k/--/MakiPan--hagrid-hand-enc-250k/train/96/conditioning_image/image.jpg" alt="Routing"> <figcaption>Conditioning Image </figcaption> </figure>

• The last option was to use <a href="https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html">MediaPipe Holistic</a> to provide pose face and hand landmarks to the ControlNet. This method was promising in theory, however, the HaGRID dataset was not suitable for this method as the Holistic model performs poorly with partial body and obscurely cropped images.

We anecdotally determined that when trained at lower steps the encoded hand model performed better than the standard MediaPipe model due to implied handedness. We theorize that with a larger dataset of more full-body hand and pose classifications, Holistic landmarks will provide the best images in the future however for the moment the hand-encoded model performs best. This repo contain the weight of Encoded hand ControlNet model

Dataset

130k image Dataset for Hand Encoding Mode

Examples

These are controlnet weights trained on runwayml/stable-diffusion-v1-5 with new type of conditioning. You can find some example images in the following.

prompt: a man in a colorful shirt giving a peace sign in front of a rallying crowd prompt: a police officer signaling someone to stop in a park