LoRA_training_sneak_peak

This is NOT A GUIDE for LoRA training, and should NEVER be treated as.

This serves as a ongoing documentation that I share some of the personal opinions or interesting findings where I learned or discovered during my journey in LoRA training.

To start with, LoRA training could be easy and quick with simple dataset preparation and training setup; but surely that's not what I'm satisfied with.
For me, the goal of making a LoRA is always 'easy to use for everyone,' which is why almost all of my character LoRAs require fewer than 10 trigger words to recreate a character.
Based on my training experience, preparing a LoRA's dataset properly is more important than using complicated and fancy training techniques. Just like cooking, you can't make any tasty dishes without using materials with good quality and prepare them properly.

Dataset Preparation

In this section, I'll outline the steps I typically follow to prepare the dataset for a LoRA model. I'll skip the "How to Get Image Data" part since there are numerous tools and methods available for obtaining image data.

1. Remove Non-Static Image Files:
   After gathering all the necessary image data for a LoRA, remove all non-static image files (e.g., .mp4, .wav, .gif, etc.). Then, convert all non-.png format images into .png.

2. Remove Duplicates:
   Take some time to go through the images and remove any duplicates. This is particularly important for large datasets, as duplicates can occur frequently.

3. Upscale Low-Resolution Images:
   Upscale all low-resolution images to at least 2k resolution. This step is crucial for improving image quality and ensuring better performance during training. Let's consider an example:

   • We have two identical PNG images, one with dimensions 907x823 pixels and the other with dimensions 3624x3288 pixels, achieved by a simple 4x upscale from SD WebUI.

   • Now, let's crop them to roughly 1k resolution (960x832 pixels in this case) using SD WebUI's Auto-size Crop:

   • Notice the difference? The image with 4x upscale has sharper edges compared to the one without any upscale, resulting in a clean and less blurry appearance when cropped into 1k resolution.

   • So, why is this important for LoRA learning?
     Inside the LoRA training script, there's a function called bucket that automatically scales up/down your image data to a resolution bucket close to the training resolution specified in the training setup.
     In this case, the pattern inside a clean image could be learned quickly; and for a somewhat low-resolution image that comes in the first place, upscaling helps sharpen the blurry lines so that the object inside is more easier to be recognized.

   • But what's the downside?
     During the process of upscaling, all the script does is essentially multiply each pixel in the original image by a certain factor (this may vary slightly depending on the upscaler). This means the upscaler doesn't actually enhance the 'details'—it simply ensures that when you zoom in, the image no longer appears blurry.
     To make matters worse, when we upscale a relatively low-resolution image (for example, 800x600 pixels) to a high resolution like 4K, we lose a significant amount of detail (such as thin lines and tiny objects) from the original image.

4. To be Continued ...

Tagging / Captioning

In this section, I'll share my approach to tagging the dataset and my opinions on tagging for character LoRAs.
Note that my discussion on this topic is based on the plugins Tagger and Dataset Tag Editor, and I'll primarily focus on the topic of character LoRA.
Please also note that much of what I say in this section is subjective, so use your own judgment if you plan to learn anything from it.

1. Tagger

   • Interrogator:
     I have used WD14 ViT (V1/V2), WD14 SwinV2 v1, and WD14 ConvNeXTV2 v1 as my tagging interrogator in the past.
     And my pick for these interrogators are WD14 ConvNeXTV2 v1 > WD14 SwinV2 v1 > WD14 ViT (V2) > WD14 ViT (V1) based on their performance in object identification.

   • Tags I'll include before tagging:
     The Additional Tags would vary depend on what kind of LoRA that we are making.
     For a character LoRA, I'd put the name of the character; for a concept LoRA, I'd put the name of the concept; etc.

   • Weight threshold: Based on my experience, a range between 0.35 - 0.4 is good enough for tagging any types of LoRA.

2. Tag Editing (Character LoRA)

   • First Step:
     When it comes to character LoRA, my initial approach is always to remove all character identity tags related to the character's face. For example: bangs, black hairs, red eyes, ponytail.
     But why?
     Let’s use common sense. What usually comes to mind first when discussing a character, such as Asuka Soryu from the anime Evangelion? The answer is obvious: your brain immediately retrieves her appearance from memory. We remember characters by their overall look, not by each individual trait, and the same goes for a character LoRA.
     Now let's go back to the question from a technical perspective:
     The point of using a character's name as an additional tag, and removing identity tags, is to embed all those small details into the tag, so that neither we nor the user have to type numerous trigger words to accurately recreate the character's face.
     To make this even more easy to understand, we are asking the LoRA to learn the fact of " a girl with a name Asuka Soryu and she looks like this", rather than "a girl with long orange hair and blue eyes and bangs has a name Asuka Soryu". See the differences? We can use "1girl, Asuka Soryu" as trigger words to recreate Asuka's face without typing "1girl, Asuka Soryu, long hair, orange hair, blue eyes, bangs", which in my opinion, is rediculous (Imagine somebody calling you in real life with "Hey [Insert your name], who has black short hair, parted bangs, and blue eyes, come here." lmao).

3. To be Continued ...