qwen3-0.6b-4b-adapter

Qwen3-0.6B text encoder adapter for FLUX.2-klein-4B

Replace klein's 4B text encoder with a 0.6B one — 7.5 GB VRAM and ~0.3 s/step less.

Train status: trained on one Vast.ai GPU (RTX 5090, 32 GB), text only — no images, no VAE, no diffusion.

Разбор обучения по-русски, с цифрами и историей решений: TRAINING.ru.md

support train please..

• Adapter: point-wise MLP 6144 -> 8192 -> 8192 -> 7680 (180.4M) plus a 2-block residual attention branch (39.6M) — adapter_v14_bal.safetensors
• Trained in two stages: the point-wise mapping first, then the attention branch — its output projection starts at zero, so the first stage is never lost
• v14 is trained on a balanced pool (3000 captions per source group: prose datasets + danbooru tags, 8 epochs, cosine LR) and with 5% empty prompts, which keeps the unconditional branch usable for CFG
• Small encoder: Qwen3-0.6B (28 layers, hidden 1024), hidden states of layers 2,9,14,18,23,27 → 6144
• Target: klein's own text_encoder (Qwen3-4B, 36 layers, hidden 2560), layers 9,18,27 → 7680
• Conditioning length 256; per-token error on held-out captions 21.7% (v11: 23.7%, first point-wise release: 24.4%)
• Works with FLUX.2-klein-4B and FLUX.2-klein-base-4B (7680 = joint_attention_dim); the distilled model wants 4 steps and guidance_scale=1.0, the base one 50 steps and guidance_scale=4.0 (CFG works there)

Before / after

Same seed, same settings, same prompt — klein with its native Qwen3-4B encoder on the left, klein driven by Qwen3-0.6B + this adapter on the right. Pose, light, clothing and the rendered sign text all carry over.

Anime and danbooru tags

Both prose and danbooru-style tag prompts work — a tag prompt (top) and an anime-style prompt (bottom), native encoder vs. this adapter.

What it is

klein's DiT conditions on 7680-dim text embeddings produced by its own encoder, a Qwen3-4B (7.5 GB in bf16). This adapter maps the embeddings of Qwen3-0.6B into that same space, so the DiT can be driven by the small encoder — the DiT, VAE and scheduler stay untouched:

text -> Qwen3-0.6B (frozen) -> hidden states of layers 2,9,14,18,23,27 -> concat (B, L, 6144)
     -> adapter -> (B, L, 7680) -> klein DiT

     y_i = MLP(x_i)  +  Attention(x)_i
           point-wise,   residual context branch:
           no context    tokenizers split "bulldog" into "Ġbulld"+"og",
                         and a point-wise map cannot compose the two

Trained on text only; the target is the reference encoder's own output for the same text. The Qwen3-0.6B weights are stock — nothing was fine-tuned. Training code: src/train_adapter.py.

Inference

python3 example.py "A red fox walks through a snowy forest at dusk." fox.png

example.py is self-contained: copy it next to adapter_v14_bal.safetensors and run. It loads Qwen3-0.6B, the adapter and klein, replaces pipe.encode_prompt and writes a 1280x768 image. The DiT, VAE, scheduler and tokenizer stay stock klein; the adapter runs in fp32. To plug the same conditioning into your own pipeline, use src/klein_condition.py.

ComfyUI

A drop-in ComfyUI workflow that drives klein with Qwen3-0.6B + this adapter instead of the native Qwen3-4B encoder: comfyui/flux2_klein_qwen3_06b_adapter.json. Custom node (the two KleinAdapter nodes) + install/run instructions: github.com/recoilme/klein-qwen3-adapter-comfyui (or see comfyui/README.md).

Limitations

• Fine identity attributes still drift: the bulldog above comes out as a terrier-like dog, although scene, light, pose and clothing stay faithful. Numerically the adapter keeps improving on these prompts (a bulldog 29.2% → 26.6% per-token error, striped bowtie 29.4% → 25.2%), but the breed itself is not transferred yet.
• Trained on prose captions and danbooru-style tags, max_sequence_length 256.
• Not needed at all if you can afford running klein's own Qwen3-4B encoder.

Files

adapter_v14_bal.safetensors  the adapter (220M: 180.4M point-wise MLP + 39.6M attention, fp32,
                             schema metadata inside)
example.py                   diffusers usage example
src/adapter_lib.py           adapter schema + loading
src/klein_condition.py       encode_prompt replacement for the klein pipeline
src/train_adapter.py         training (online encoders, no cached embeddings)
src/export_safetensors.py    .pt -> .safetensors with the schema metadata inside
TRAINING.ru.md               detailed Russian write-up: how it was trained and why
media/before_after.png       main illustration: native Qwen3-4B vs Qwen3-0.6B + adapter
media/tags_before_after.png  danbooru tags and anime-style prompts
media/limitations.png        known issue: object identity (bulldog)

Donations

Thanks for your support!

• Euro / dollars: sorry, we can’t accept payments via Patreon or Ko-fi. Please register on vast.ai (this is the GPU provider we use for training the model), top up any amount, and transfer: Transfer Money / User / Email: [email protected].
• RUB: donate in rub
• DOGE: DEw2DR8C7BnF8GgcrfTzUjSnGkuMeJhg83
• BTC: 3JHv9Hb8kEW8zMAccdgCdZGfrHeMhH1rpN
• USDT
  • Ethereum / Polygon / BNB SmartChain: 0xD4388B6698dFaE1460E72099D4F208aaCA4f6E6C
  • Tron: TD7ey4h9igPGdcrcBcnZaz56R5tNgRZNvV
  • Solana: MMYFJeYEtYHrSNFHChytJDHbEDniXrnAxPNLhJ1LbkB

Contacts

• telegram recoilme *prefered way
• mail at aiartlab.org (slow response)

Citation

@misc{klein_qwen3_adapter,
  title={qwen3-0.6b-4b-adapter: Qwen3-0.6B text encoder for FLUX.2-klein-4B},
  author={recoilme and AiArtLab Team},
  url={https://huggingface.co/AiArtLab/qwen3-0.6b-4b-adapter},
  year={2026}
}