ClipProj-MiniMax-H3

ClipProj — MiniMax H3 conditioning from a Qwen3-VL-4B

Projection matrices that let a Qwen3-VL-4B replace the Qwen3-VL-32B text encoder of MiniMax H3.

15.7 GB → 5.2 GB of VRAM, with no change to the diffusion model, the VAEs or the sampler.

⚠️ Proof of concept — working, but a proof of concept. It runs and produces good video, and every number below was measured on real hardware. Built and tested on a single setup (Windows 11, NVIDIA, ComfyUI 0.31.0) with deliberately limited exploration.

These files are useless on their own. They require the custom node: github.com/nicolab28/ComfyUI-ClipProj

Now in safetensors

Every matrix is available as .safetensors. Prefer it. A .pt goes through pickle, which can execute arbitrary code the moment it is opened — an absurd risk for files that hold nothing but tensors and a handful of scalars. The scalars now live in the safetensors header and the node converts them back on load.

Reading them needs node v0.1.2. The .pt files are kept alongside for now so that anyone still on an older node has something that works; they will go away in a later release.

Requested by @lucas-ai26, who was right.

Re-download if you got the first release

The first release was missing the attention-sink vector, and short prompts suffered for it. The first token of a sequence is an attention sink: its direction is constant from one prompt to the next (cosine 1.0000 measured over 1966 prompts) and it carries nothing from the text, yet its norm reaches 16 500 against 291 for a text token. Calibration excluded it, rightly — its extreme values would wreck the statistics — but the node projected it anyway, through a matrix that had never seen one, producing an arbitrary vector of enormous norm. Invisible on a 200-token prompt where it is 0.5 % of the positions, ruinous on a 7-token one where it is 14 %.

Every matrix here now carries the measured sink vector and substitutes it at inference. Since the vector is constant, this is not an approximation: it is the value the 32B would have produced.

Also new in this release: an 8B matrix, an int8_convrot matrix, and ref2va now works — see below.

What this is

MiniMax H3 conditions on a Qwen3-VL-32B truncated to 50 layers — 15.7 GB in NVFP4 — solely to turn a prompt into a [seq, 5120] tensor. This repository provides a learned linear map that lets a much smaller Qwen3-VL produce the same conditioning:

cond = ((h - mean_in) / std_in) @ W * std_out + mean_out

It works because every Qwen3-VL shares the same tokenizer (151936 tokens): a prompt yields the same tokens at the same positions in both models, so a position-by-position mapping between their hidden states can be learned. Calibration is plain ridge regression — no gradients, no epochs, no learning rate.

Files

Put them in ComfyUI/models/clip_projections/.

Take the four files at the root. Everything in obsolete/ is superseded and kept only so the comparison can be reproduced.

* measured in a different space from the rows below — the two sets of numbers cannot be compared. See the next section.

<details> <summary><code>obsolete/</code> — the unweighted matrices, kept for comparison</summary>

These were the v0.1.1 release. They work, and the 2 000-prompt one is calibrated on eight times more data than anything at the root — but they minimise the wrong error, and the root matrices are the ones that generation testing actually ran on. Keep them if you want to A/B the weighting yourself.

</details>

The 8B matrices need an 8B encoder — different input width, 4096 instead of 2560, so they are not interchangeable with the 4B ones. The node checks the width and refuses a mismatch.

Every matrix here works on any variant of its own size: the measured cosine gap between a bf16-calibrated matrix applied to an abliterated fp8 encoder is 0.0023. You do not need the exact checkpoint a matrix was calibrated on.

The CONDPROJ matrices

These are the ones the day's generation testing actually ran on, including ref2va, and they are the recommended starting point.

The DiT does not consume the conditioning as it arrives: it first passes it through condition_proj, a Linear(5120 → 5376) feeding the token refiner. That layer's spectrum is very uneven — a factor of 45 between the top and bottom deciles of its singular values, and 52 % of the energy in 10 % of the directions. Plain ridge regression ignores this and spends as much effort reconstructing a direction the DiT will multiply by 0.10 as one it will multiply by 37.

The CONDPROJ matrices are calibrated against the output of that layer instead of its input, so the error being minimised is the error the DiT will actually see. The result is then mapped back to 5120 dimensions through the pseudo-inverse — the layer is injective, so the round trip loses nothing. The output is still a standard conditioning tensor and the diffusion model is not modified. Only the error metric changed.

Their cosine is measured in the 5376-dimensional space, which is why it cannot be compared with the unweighted figures. Judge them by generation, not by the number.

Credit for the idea goes to u/stddealer on r/StableDiffusion, who pointed out that condition_proj was sitting there.

Measured results

Eight times the data buys 1.8 % of cosine: the linear projection is at its ceiling, not starved of data. Going further needs an MLP, not more prompts.

A cosine of 0.71 sounds poor and is not — the DiT tolerates far more than the metric suggests. What holds up in actual generation: simple prompts, structured multi-shot prompts (four distinct cuts with no bleed between them), fl2va with first and last frame, and ref2va with a reference image.

Fidelity does not collapse on short prompts, contrary to what one might expect from a corpus of long ones: measured per-token cosine goes from 0.937 at 80 words down to 0.908 at 2 words, a 3 % decline, once the sink is handled.

Run the controls first

The two control matrices exist to prove the learned matrix is doing the work rather than the diffusion model. Same prompt, same seed, only the matrix changes:

‖W_identity‖ = 50.6 against ‖W_learned‖ = 52.4 — near-identical energy, so the difference is structural, not a matter of scale.

If the identity control ever looks fine, the learned matrix adds nothing — and you want to know that before trusting it.

Known limitations

Named references depend on the encoder, not on the projection. This was measured rather than guessed: asking the encoder to describe a person in plain text — which bypasses the matrix entirely — shows where the knowledge actually stops. The 4B places Scarlett Johansson correctly as Black Widow but believes she has dark brown hair; the 8B describes her correctly as blonde with blue eyes. So when a proper noun renders as the wrong person, question the encoder first.

The practical workaround is to describe rather than name: "the actress X as [role], blonde, ..." recovers an identity that the bare name loses, on both the 4B and the 8B. A name is a fragile signal carried by two or three tokens; a description spreads it over a dozen redundant ones and the reconstruction error averages out instead of accumulating.

Note that this does not apply to ref2va: there the identity comes from the reference image, so the encoder's factual gaps stop mattering.

Quantisation costs facts. The same test on int8_convrot against bf16 shows two factual errors appearing under quantisation. Fine for general use, worth knowing if your prompts lean on proper nouns.

Speech in languages other than English degrades. Reported and reproduced with French: the 32B pronounces it cleanly, a projected 4B or 8B does not. This is not a corpus problem — measured on identical English prompts differing only in the quoted line, French tokens reconstruct at 0.8974 against 0.8996 for English, which is noise. A cosine of 0.90 is ample for visual semantics and insufficient for phonetics; the DiT's audio branch is far more demanding than its image branch, and a language the model handles less confidently has less margin to absorb the error.

Required models

The 32B text encoder is no longer needed — that is the entire point.

Licence and responsibility

These matrices are released under MIT, like the node.

They are derived from the activations of both models, and their legal status is unclear. They are provided as-is, for research, with no claim of ownership over anything derived from the underlying models.

• Qwen3-VL is published by Alibaba under Apache 2.0. Read and comply with its terms and acceptable-use policy.
• MiniMax H3 ships under a custom licence. Read it before any use, particularly commercial.

This project is not affiliated with, endorsed by, or connected to Alibaba / Qwen, MiniMax, or Comfy Org.

You remain responsible for what you generate and for complying with the licences of every model you load.

Credits

Vibe-coded with Anthropic Claude Code (Opus 5). Every number quoted was measured on real hardware, not estimated: where a prediction turned out wrong, the measurement won and the text was corrected.