flux-imatrix

Supported?

Expect broken or faulty items for the time being. Use at your own discretion.

• ComfyUI-GGUF: all? (CPU/CUDA)
  • Fast dequant: BF16, Q8_0, Q5_1, Q5_0, Q4_1, Q4_0, Q6_K, Q5_K, Q4_K, Q3_K, Q2_K
  • Slow dequant: others via GGUF/NumPy
• Forge: TBC
• stable-diffusion.cpp: llama.cpp Feature-matrix
  • CPU: all
  • Cuda: all?
  • Vulkan: >= Q3_K_S, > IQ4_S; PR IQ1_S, IQ1_M PR IQ4_XS
  • other: ?

Disco

Dynamic quantization:

• time_in.in_layer: Q8_0/Q6_K
• final_layer, vector_in.in_layer, guidance_in: Q8_0
• vector_in.out_layer, time_in.out_layer, txt_in, img_in: F16
• single_blocks.[> 10 && < 37].modulation.lin: one down?

Caesar

Combined imatrix multiple images 512x512 and 768x768, 25, 30 and 50 steps city96/flux1-dev-Q8_0 euler

data: load_imatrix: loaded 314 importance matrix entries from imatrix_caesar.dat computed on 475 chunks

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Dynamic quantization:

• img_in, guidance_in.in_layer, final_layer.linear: f32/bf16/f16
• guidance_in, final_layer: bf16/f16
• img_attn.qkv, linear1: some layers two bits up
• txt_mod.lin, txt_mlp, txt_attn.proj: some layers one bit down

Experimental from f16

Observations

• More imatrix data doesn't necessarily result in better quants
• I-quants worse than same bits k-quants?
• Quant-dequant loss

Bravo

Combined imatrix multiple images 512x512 25 and 50 steps city96/flux1-dev-Q8_0 euler

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Experimental from f16

Observations

• Bravo IQ1_S worse than Alpha?
• Latent loss
• Per layer quantization cost from chrisgoringe/casting_cost
• Per layer quantization cost 2 from Freepik/flux.1-lite-8B: double blocks and single blocks
• Ablation latent loss per weight type
• Pareto front loss vs. size

Alpha

Simple imatrix: 512x512 single image 8/20 steps city96/flux1-dev-Q3_K_S euler

data: load_imatrix: loaded 314 importance matrix entries from imatrix.dat computed on 7 chunks.

Using llama.cpp quantize cae9fb4 with modified lcpp.patch.

Experimental from q8

Observations

Sub-quants not diferentiated as expected: IQ2_XS == IQ2_S, IQ3_XS == IQ3_S == IQ3_M, Q3_K_M == Q3_K_L.

• Check if lcpp_sd3.patch includes more specific quant level logic
• Extrapolate the existing level logic