Downloads · 30 days
45
14% of all-time downloads
je-suis-tm/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4
wong_kar_wai_my_blueberry_nights_lora_flux2_nf4 is a text-to-image model from je-suis-tm. Use it when you need an image from a text prompt. It is set up for diffusers. The card lists the license as mit.
Downloads · 30 days
45
14% of all-time downloads
All-time downloads
315
Public
Repo size
15.2 MB
Likes
0
Public
Click a slice to open those files.
.png8.9 MB · 58%
From the Hugging Face model README
All files are also archived in https://github.com/je-suis-tm/huggingface-archive in case this gets censored.
Wong Kar Wai is one of my fav directors. This QLoRA intends to replicate that grainy, saturated jewel tone, dimly-lit vibe of the movie My Blueberry Nights (2007).
The training is based on https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/test_dreambooth_lora_flux2.py. Everything in this training script needs to be set at the same torch dtype as the script was designed for unquantized version which will export in float32. The training took 14 hours on A100 80GB with max VRAM consumption at 32GB. The inference consumes 35GB of VRAM. To avoid running low on VRAM, both transformers and text_encoder were quantized.
export MODEL_NAME="diffusers/FLUX.2-dev-bnb-4bit"
export INSTANCE_DIR="/pvol/wong_kar_wai_my_blueberry_nights"
export OUTPUT_DIR="/pvol/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4"
export Q_DIR="/pvol/quantization_config.json"
accelerate config default
accelerate launch train_dreambooth_lora_flux2.py \
--pretrained_model_name_or_path=$MODEL_NAME \
--mixed_precision="bf16" \
--dataset_name=$INSTANCE_DIR \
--output_dir=$OUTPUT_DIR \
--bnb_quantization_config_path=$Q_DIR \
--gradient_checkpointing \
--cache_latents \
--instance_prompt="WKW style" \
--caption_column="text" \
--aspect_ratio_buckets="800,1904" \
--center_crop \
--train_batch_size=1 \
--guidance_scale=1 \
--use_8bit_adam \
--offload \
--checkpointing_steps=100 \
--gradient_accumulation_steps=4 \
--optimizer="adamW" \
--learning_rate=1e-4 \
--lr_scheduler="constant" \
--lr_warmup_steps=100 \
--max_train_steps=1500 \
--rank=4 \
--seed="0"
import torch
from transformers import Mistral3ForConditionalGeneration
from diffusers import Flux2Pipeline, Flux2Transformer2DModel
repo_id = "diffusers/FLUX.2-dev-bnb-4bit"
device = "cuda:0"
torch_dtype = torch.float32 #only supports float32 when using train_dreambooth_lora_flux2.py
transformer = Flux2Transformer2DModel.from_pretrained(
repo_id, subfolder="transformer", torch_dtype=torch_dtype, device_map="cuda:0"
)
text_encoder = Mistral3ForConditionalGeneration.from_pretrained(
repo_id, subfolder="text_encoder", dtype=torch_dtype, device_map="cuda:0"
)
pipe = Flux2Pipeline.from_pretrained(
repo_id, transformer=transformer, text_encoder=text_encoder, torch_dtype=torch_dtype
)
pipe.load_lora_weights("je-suis-tm/wong_kar_wai_my_blueberry_nights_lora_flux2_nf4",
weight_name='pytorch_lora_weights.safetensors')
pipe.enable_model_cpu_offload()
prompt = "WKW Style, A woman in a yellow dress with a lace trimmed cowboy hat poses against a weathered brick column. tan brick columns forms the backdrop, punctuated by a glimpse of a vehicle"
image = pipe(
prompt=prompt,
generator=torch.Generator(device=device).manual_seed(42),
num_inference_steps=50, # 28 is a good trade-off
guidance_scale=4,
height=800,
width=1904, #the movie was in a weird resolution 800*1912, the resolution needs to be divided by 16 so 1904
).images[0]
image.save("wong_kar_wai_my_blueberry_nights.png")
You should use WKW style to trigger the image generation.
Download them in the Files & versions tab.