nunchaku-qwen-image-2.1

Qwen-Image-2.1 (Nunchaku SVDQuant NVFP4 r32 with Channel-Wise Macro Scaling) 🌟🚀

This repository provides the high-fidelity SVDQuant NVFP4 (Rank 32) quantized transformer for Alibaba's Qwen-Image-2.1, forged with channel-wise Level-0 macro scaling (wcscales) for inference with Nunchaku on NVIDIA Blackwell (sm_120), Ada Lovelace, and Hopper architectures.

---

⚡ Highlights

• 100% Resident VRAM Execution on 16GB GPUs:
  • Weights compressed from 30.2 GB (BF16) down to 4.08 GB (NVFP4) — a 7.4x reduction in transformer memory footprint!
  • Runs entirely resident in VRAM on consumer GPUs (e.g. NVIDIA GeForce RTX 5060 Ti 16GB, RTX 4080 16GB, RTX 4090 24GB) with zero layer-by-layer PCIe streaming bottlenecks.
• Channel-Wise Level-0 Macro Scaling (wcscales):
  • Unlike standard scalar-scale quantization that suffers from SECAM-like chroma bleeding and dynamic range clipping, this checkpoint incorporates 224 channel-wise scale vectors ([4096] dimensions).
  • Boosts PSNR by +4.62 dB over scalar-scale quantization and raises Structural Similarity (SSIM) from 0.84 to 0.92+ across complex photorealistic scenes and high-frequency textures.
• SVDQuant Rank-32 Error Compensation:
  • Integrates a 16-bit low-rank residual branch alongside 4-bit normal float lattice quantization across all 32 transformer blocks (attn.to_q, to_k, to_v, to_out.0, img_mlp.proj, img_mlp.gate_layer, img_mlp.out).
• Designed for Pairing with Heretic Text Encoder:
  • Perfectly complements Qwen21_Text_Encoder_Heretic for maximum prompt adherence, eliminating vector deflection and maintaining fine semantic control.
• Blazing Fast Local Inference:
  • 25-step Text-to-Image (1024x1024) in ~20.0s on a single RTX 5060 Ti (vs 64.8s on unquantized offloaded baselines — a 3.2x speedup).
  • 40-step In-Context Image Editing (1024x1024) in ~41.0s (vs 198.4s — a 4.8x speedup).

---

📊 Benchmark & Quality Evaluation (RTX 5060 Ti 16GB)

1. Latency & Resource Utilization (1024 x 1024)

2. Reconstruction Fidelity (vs Unquantized BF16 Ground Truth)

---

🚀 Quickstart Usage

1. Requirements & Setup

Ensure you have PyTorch, Diffusers, and Nunchaku installed:

pip install nunchaku diffusers transformers accelerate sentencepiece

2. Loading with Diffusers and Nunchaku

import torch
from diffusers import QwenImage21Pipeline
from nunchaku.models.transformers.transformer_qwenimage21 import NunchakuQwenImage21Transformer2DModel

device = "cuda:0"

# 1. Load forged NVFP4 DiT directly into resident VRAM
transformer = NunchakuQwenImage21Transformer2DModel.from_pretrained(
    "svdq-fp4_r32-qwen-image-2.1.safetensors",
    device=device,
    torch_dtype=torch.bfloat16,
)

# 2. Assemble complete pipeline (pointing text encoder to Heretic if desired)
pipeline = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1",
    transformer=transformer,
    torch_dtype=torch.bfloat16,
)
pipeline.vae = pipeline.vae.to(device)
pipeline.vae.enable_tiling()

# 3. Generate high-resolution image
image = pipeline(
    prompt="A serene Japanese garden with blooming cherry blossoms, koi pond, and morning mist, highly detailed",
    height=1024,
    width=1024,
    num_inference_steps=25,
    true_cfg_scale=1.0,
).images[0]

image.save("serene_garden.png")

3. Pairing with Heretic Text Encoder (Qwen21_Text_Encoder_Heretic)

When generating scenes with complex human anatomy, athletic dynamics, or specific attire, pair this DiT with Qwen21_Text_Encoder_Heretic to eliminate VLM hesitation vector deflection and ensure exact prompt adherence:

# Launch high-throughput server with Heretic text encoder on port 4500:
./extras/imagegen_qwen21_nvfp4.sh 4500

---

📦 Packaged Sources & Tooling (extras/)

Inside the extras/ folder of this repository, you will find:

• ImageEditServer.py: High-concurrency FastAPI/Uvicorn server supporting Text-to-Image (/v1/images/generations) and Multimodal In-Context Image Editing (/v1/images/edits).
• QwenImage21NVFP4Backend.py: Production backend with resident DiT management, VAE tiling, and optional PCIe layerwise text encoder streaming.
• stream_encoder.py: Zero-quality-loss layerwise weight streaming engine for Qwen3-VL-8B.
• stream_quantize_qwen21.py: Full quantization pipeline implementing alternating least-squares SVD rank-32 decomposition and channel-wise scale packing.
• diagnose_trajectory_and_scales.py: Step-by-step Euler flow-matching trajectory analyzer.
• eval_divergence_metrics.py: Multi-domain divergence evaluator (PSNR, SSIM, SigLIP cosine similarity, edge acuity, chroma spill).

---

📜 Citation & Credits

• Qwen-Image-2.1: Qwen Team, Alibaba Cloud.
• SVDQuant & Nunchaku: MIT HAN Lab (SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models).
• Quantization & Channel-Scaling Optimization: Oleg K. / Nikola Seeker Project.