wan2.2-14b-svd-merge

WAN 2.2 14B — SVD-merged

Single-base + low-rank-adapter form of the WAN 2.2 T2V-A14B mixture-of-experts model. The original WAN 2.2 ships two separate 14B dense transformers (high_noise_model and low_noise_model). Their weights differ substantially in magnitude (mean ‖W_high − W_low‖_F / ‖W_low‖_F ≈ 0.46) but the difference is highly low-rank — rank-99 of Δ is ~10% of min(m, n) per layer. This repo stores:

• base.<layer_key> — W_low (the low-noise expert; also serves as the shared base)
• delta.<layer_key>.{U,S,Vt} — SVD of (W_high − W_low) at rank q=min(min_dim, 512)
• delta_raw.<layer_key> — raw difference for 1-D / tiny tensors (biases, norms)

The full model is uploaded; consumers pick a runtime rank truncation.

Usage

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import json

state = load_file(hf_hub_download("gueraf/wan2.2-14b-svd-merge", "wan22_14b_svd_merge.safetensors"))
meta  = json.loads(open(hf_hub_download("gueraf/wan2.2-14b-svd-merge", "meta.json")).read())

# At runtime, reconstruct W_high at chosen rank r per layer:
key = "blocks.0.self_attn.q.weight"
W_low = state[f"base.{key}"]
U, S, Vt = state[f"delta.{key}.U"], state[f"delta.{key}.S"], state[f"delta.{key}.Vt"]
r = int(0.10 * meta[key]["min_dim"])             # 10% of min_dim ≈ 99% of Δ energy
W_high_approx = W_low + (U[:, :r] * S[:r]) @ Vt[:r, :]

Provenance

• Source: Wan-AI/Wan2.2-T2V-A14B (high_noise_model/wan2.2-t2v-14b-highnoise.pth, low_noise_model/wan2.2-t2v-14b-lownoise.pth), fp32.
• Decomposition: per-layer randomised SVD via torch.svd_lowrank(q=512, niter=6) — q=512 captures the full useful spectrum since rank-99 of Δ is ~10% of min_dim (~358 for 14B).
• Storage dtype: fp32 throughout (matches input dtype).

Spectral notes

• WAN 2.2 14B High Noise: α_w = 3.49 (well-trained ✓)
• WAN 2.2 14B Low Noise: α_w = 3.97 (well-trained ✓)

See the analysis report for the full WeightWatcher analysis and the SVD foldability finding (Finding 5).