Flux2-Klein-9B-True-V3-fp8

Flux2-Klein-9B-True-V3-fp8

An FP8 build of wikeeyang/Flux2-Klein-9B-True-V3, saved in the optimum-quanto QuantizedDiffusersModel layout so it loads directly into a diffusers Flux2KleinPipeline with no conversion step.

This is a format and precision conversion only. Nothing was retrained, distilled or merged. The weights are wikeeyang's True-V3.

⚠️ Licence — read before use

These weights derive from black-forest-labs/FLUX.2-klein-9B, released under the FLUX Non-Commercial License v2.1. Commercial or revenue-generating deployment requires a separate licence from Black Forest Labs — see bfl.ai/licensing.

Only FLUX.2-klein-4B is Apache-2.0. The 9B models are not. The upstream True-V3 repository is tagged apache-2.0; that tag does not reflect the terms of the base model, and a third party cannot relicense weights they do not own. This repository is labelled to match the actual upstream terms.

Why this exists

True-V3 ships ComfyUI single-file formats only — bf16, fp8mixed, int8mixedrow, GGUF, nvfp4, int4_convrot. None of them is a quanto checkpoint: QuantizedDiffusersModel.from_pretrained() expects quanto's own directory layout (_data/_scale tensors plus a quanto config), and quanto cannot read ComfyUI's scaled-FP8 tensor layout.

So a diffusers/quanto serving stack could not use True-V3 without converting it first. This repository is that conversion, published so nobody has to redo it.

Usage

import torch
from diffusers import Flux2KleinPipeline, Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel

# quanto's Marlin FP8 kernel rejects a non-contiguous activation, which the
# Flux2 attention path produces. Apply before loading.
from optimum.quanto.nn.qlinear import QLinear
_orig = QLinear.forward
QLinear.forward = lambda self, x: _orig(self, x.contiguous())


class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel


pipe = Flux2KleinPipeline.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B",
    transformer=None,
    torch_dtype=torch.bfloat16,
).to("cuda")

pipe.transformer = QuantizedFlux2KleinTransformer.from_pretrained(
    "ModelsLab/Flux2-Klein-9B-True-V3-fp8"
).to("cuda")

image = pipe(
    prompt="a vintage storefront window with a hand-painted gold sign that reads "
           '"KLEIN COFFEE ROASTERS EST 1974", rainy street reflection',
    height=1024, width=1024,
    num_inference_steps=4,
    generator=torch.Generator("cuda").manual_seed(1234),
).images[0]
image.save("out.png")

Tighter VRAM

The transformer is 9.08 GB. The bf16 text encoder is another ~16 GB, which will not fit alongside it on a 24 GB card. Quantise it the same way:

from optimum.quanto import QuantizedModelForCausalLM, qfloat8
from transformers import AutoModelForCausalLM

te = AutoModelForCausalLM.from_pretrained(
    "black-forest-labs/FLUX.2-klein-9B", subfolder="text_encoder",
    torch_dtype=torch.bfloat16)
pipe.text_encoder = QuantizedModelForCausalLM.quantize(
    te, weights=qfloat8).to("cuda")

FP8 transformer + FP8 text encoder is ~18 GB resident. Measured peak allocation on an RTX 3090 at 1024×1024, 4 steps, with both resident: 20.66 GB — so it fits a 24 GB card, with roughly 3 GB spare.

Measured against the stock FP8 checkpoint

Same prompt, same seed, same 4 steps, same quantiser. Top row is the stock FP8 checkpoint, bottom row is this one. Note the misspelled "COFFFEE" in the stock render.

Both checkpoints were run through an identical harness: same architecture, same quantiser, same 4 steps, same guidance, same seeds, and prompt embeddings encoded once and cached so both saw bit-identical conditioning. 12 prompts × 3 seeds = 36 timed renders each, 1024×1024, on an RTX 3090.

¹ CLIP image–image cosine ×100 between each edit and its reference images, over a two-reference editing run (four scenes × two seeds), scored on the render set shared by every variant tested.

The latency difference is inside run-to-run noise — across eight variants benchmarked the same way, the whole spread was 0.06 s. Same architecture, same step count, same FP8 kernels, so this is a quality change at fixed cost rather than a trade.

Qualitatively, the clearest wins over stock are rendered text (the sign prompt above is misspelled "COFFFEE" by the stock checkpoint and correct here), hands, and product/material detail. Colour is slightly less saturated and more filmic.

Caveat on the metrics: the LAION aesthetic predictor rewards contrast and saturation, so it is not a neutral judge of photographic realism — a crunchier model can score well while rendering worse text. Treat the table as a ranking aid and look at your own prompts.

Verification

This checkpoint was checked by regenerating four benchmark prompts at a fixed seed and diffing them against the renders produced by the original bf16 True-V3 weights in the earlier benchmark run:

Bit-identical on all four. The conversion changes the container, not the output.

How this was built

Reproducible in about a minute of compute once the 18 GB source file is local (63 s measured: load, quantise, save):

import torch
from diffusers import Flux2Transformer2DModel
from optimum.quanto import QuantizedDiffusersModel, qfloat8
from huggingface_hub import hf_hub_download


class QuantizedFlux2KleinTransformer(QuantizedDiffusersModel):
    base_class = Flux2Transformer2DModel


path = hf_hub_download("wikeeyang/Flux2-Klein-9B-True-V3",
                       "Flux2-Klein-9B-True-V3-bf16.safetensors")

tr = Flux2Transformer2DModel.from_single_file(
    path,
    config="black-forest-labs/FLUX.2-klein-9B", subfolder="transformer",
    torch_dtype=torch.bfloat16,
)
QuantizedFlux2KleinTransformer.quantize(tr, weights=qfloat8).save_pretrained("out")

Notes and caveats

• Ampere (RTX 3090 / A100): these cards have no native FP8 tensor cores. quanto routes through its Marlin FP8 GEMM, which does run on sm_80/sm_86 — so FP8 here buys memory, and the speed is comparable to the equivalent bf16 path rather than dramatically faster. Ada and Hopper get more from it.
• The contiguity patch in the usage snippet is required, not optional. Without it the Marlin kernel raises RuntimeError: A is not contiguous on the first forward pass.
• LoRAs must be fused before quantisation. Calling load_lora_weights() on an already-quantised transformer does not work. Load the bf16 transformer, fuse_lora(), then quantise.
• Only the transformer is replaced. Text encoder, VAE, scheduler and tokenizer come from the base FLUX.2-klein-9B repo unchanged.
• guidance_scale is ignored. Klein is step-distilled, and diffusers emits "Guidance scale 4.0 is ignored for step-wise distilled models" and drops the value. Passing it does nothing; do not expect it to behave as a tuning knob.

Credits

• wikeeyang — the True-V3 finetune. All of the quality improvement is theirs; this repository only changes the container.
• Black Forest Labs — FLUX.2-klein-9B.
• optimum-quanto — the quantiser and the checkpoint format.

Converted and published by ModelsLab.