flux.1-dev-nunchaku-lite-int4-r32

FLUX.1 Dev Nunchaku Lite INT4 r32

Diffusers-loadable conversion of:

• Base model: black-forest-labs/FLUX.1-dev
• Source repo: nunchaku-ai/nunchaku-flux.1-dev
• Source checkpoint: svdq-int4_r32-flux.1-dev.safetensors

The transformer uses quant_method: nunchaku_lite, INT4 SVDQ with group size 64, runtime rank 64, 418 SVDQ targets, and 76 AWQ W4A16 targets. The CLIP encoder is copied from the base model and T5 text_encoder_2 is BitsAndBytes 4-bit NF4. Fused QKV modules are split in logical tensor layout; single-block proj_out is merged from attention and MLP projections; low-rank tensors are logically padded to rank 64. INT4 shifted down-projection biases are compensated for signed-unfused Diffusers execution.

Benchmark

RTX 5090, 1024×1024, 28 steps, guidance scale 3.5, one warmup and three measured runs, full GPU placement. VRAM is peak total device usage sampled with nvidia-smi, including allocations outside PyTorch's caching allocator.

Output Comparison

Both images use the same prompt, seed 0, scheduler, resolution, and step count. Native Nunchaku 1.x refuses INT4 checkpoints on Blackwell GPUs, so a same-precision native benchmark was unavailable on the RTX 5090.

Run

Requires the Hugging Face kernels package and a Turing, Ampere, Ada, or Blackwell NVIDIA GPU; Hopper is unsupported for INT4 kernels.

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
    "lite-infer/flux.1-dev-nunchaku-lite-int4-r32",
    torch_dtype=torch.bfloat16,
).to("cuda")

image = pipe(
    prompt='A cinematic photograph of a red fox standing in a misty forest at sunrise, detailed fur, volumetric light',
    generator=torch.Generator("cuda").manual_seed(0),
    width=1024,
    height=1024,
    num_inference_steps=28,
    guidance_scale=3.5,
).images[0]
image.save("output.png")