ERNIE-Image-Turbo-SDNQ-uint4-static

ERNIE-Image-Turbo SDNQ UINT4 Static

This is a 4-bit SDNQ static quantization of baidu/ERNIE-Image-Turbo. The published SDNQ configs set use_quantized_matmul=true for pe, text_encoder, transformer, and the pipeline-level config. For current SDNQ/Diffusers builds, enable quantized matmul explicitly after loading with apply_sdnq_options_to_model; the serialized flag is retained in metadata, but may not be applied automatically by from_pretrained().

Recipe

• Base model: baidu/ERNIE-Image-Turbo
• Quantizer: sdnq / SDNQ UINT4 static, dequantize_fp32=false
• Quantized components: pe, text_encoder, transformer
• Runtime validation: use_quantized_matmul=true
• Validation GPU: NVIDIA RTX 6000 Ada Generation
• Validation settings: 10 fixed prompt/seed pairs, 8 inference steps, guidance scale 1.0, use_pe=False
• Runtime note: do not set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:32 for this pipeline; it caused allocator over-reservation and much slower denoising in validation.
• Machine-readable runtime recommendations are stored in runtime_config.json.

use_pe=False is used for the headline validation table to compare the image models directly. Stage-level debugging showed that use_pe=True can dominate latency: on the 1200x896 technical-diagram prompt, pe.forward accounted for most of the runtime, while the denoising transformer was much smaller.

Measured Results

The row above is preserved for reproducibility of the original validation run. A follow-up profiling pass found that current loaders may leave quantized matmul disabled unless it is applied explicitly after loading.

Explicit Quantized-Matmul Runtime

With explicit apply_sdnq_options_to_model(..., use_quantized_matmul=True), default PyTorch CUDA allocator settings, and no torch.cuda.empty_cache() between hot generations:

The slow component with PE disabled is the denoising transformer. In the corrected qmm profile, transformer.forward accounts for roughly 5.0-5.4s of a 5.8-7.0s hot generation on RTX 6000 Ada. text_encoder.forward is about 0.55-0.65s after warmup, and vae.decode is usually about 0.15s.

The allocator pitfall is large: with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:32, the same explicit-qmm runtime reserved about 48 GiB and measured 25.88s hot median with empty_cache=True, or 15.86s without empty_cache.

Visual Comparison

Individual prompt pairs are stored in comparison/, and full metrics are stored in metrics/.

Usage

import torch
import sdnq  # registers SDNQ support
from diffusers import ErnieImagePipeline
from sdnq.loader import apply_sdnq_options_to_model

pipe = ErnieImagePipeline.from_pretrained(
    "WaveCut/ERNIE-Image-Turbo-SDNQ-uint4-static",
    torch_dtype=torch.bfloat16,
).to("cuda")

for name in ("pe", "text_encoder", "transformer"):
    component = getattr(pipe, name, None)
    if component is not None:
        setattr(pipe, name, apply_sdnq_options_to_model(component, use_quantized_matmul=True))

image = pipe(
    prompt="A clean modern poster with readable Cyrillic typography",
    width=1024,
    height=1024,
    num_inference_steps=8,
    guidance_scale=1.0,
    use_pe=False,
).images[0]

If you need maximum throughput, keep the model resident and avoid calling torch.cuda.empty_cache() between requests.

You can confirm the runtime state after loading:

for name in ("pe", "text_encoder", "transformer"):
    qcfg = getattr(getattr(pipe, name, None), "quantization_config", None)
    print(name, getattr(qcfg, "use_quantized_matmul", None))

Prompt Set

Notes

• The comparison uses the same prompts, dimensions, seeds, 8 inference steps, and guidance scale for both original and quantized runs.
• use_pe=True remains supported by the pipeline, but it measures prompt-enhancer behavior in addition to image generation.
• Corrected qmm runtime metrics are stored in metrics/ernie_uint4_qmm_explicit_default_allocator_8step_metrics.json; allocator-debug metrics are stored in metrics/runtime_allocator_debug_metrics.json.
• This is an independent quantized artifact; see the original Baidu model card for upstream model details, benchmarks, and license terms.