WAN-Fun-LoRA-5B-Control-TrainCFG4-4Step-CFG1-step1600

WAN Fun LoRA: Train CFG4 (RGS3), step 1600, 4-step inference CFG1

This is the training-step-1600 generator LoRA distilled directly on alibaba-pai/Wan2.2-Fun-5B-Control using prompt + dynamic Depth video pairs. It is intended for four denoising steps with guidance_scale=1.0, which disables classifier-free guidance. The training teacher uses CFG=4 (real_guidance_scale=3, RGS3); TrainCFG4 in the repository name records that training setting.

中文：这是专门在 Wan2.2-Fun-5B-Control 上训练的 DMD LoRA。发布训练 1600 step 检查点；推理 4 steps、CFG=1（关闭 CFG）。它就是本次 PAI-Bench-C Depth 评测使用的权重。训练时 generator、teacher、critic 均加载 完整 Fun-Control 模型；发布文件仅包含推理所需的 generator LoRA。

step1600 describes the saved training iteration. 4Step describes inference. CFG and LoRA strength are separate settings. The LoRA residual is strength * (alpha/rank) * (B @ A).

During DMD training the teacher used real_guidance_scale=3, with the formula conditional + 3 * (conditional - unconditional). This corresponds to a conventional teacher CFG scale of 4, while the student/critic branch and the released inference recipe use conventional CFG 1.

Training and provenance

The run used two nodes with eight H100 GPUs each, microbatch one per GPU, gradient accumulation two, and effective global batch 32. It reused the exact 5,000 prompt + dynamic-depth pairs from the preceding ControlNet Depth run:

• Depth videos: Yanbin99/Depth-Normal-Videos-42K, revision 9c5d205edd71077bbab7d96cae971a25a1fbf91f.
• Corresponding captions: fal/cosmos-openvid-1m, revision 10b41fc29006eff62ff64b8795b8ae8ef7ff9cde.
• Training geometry: 832 x 480, 93 raw video frames, 24 latent frames.
• Generator, frozen teacher, and trainable critic each load the full Fun-Control DiT.
• Generator and critic have independent rank-64 adapters; the teacher is frozen.

The published adapter contains 600 FP32 tensors over 300 Linear layers in 30 blocks: self-attention q/k/v/o, cross-attention q/k/v/o, and FFN 0/2. There are 161,218,560 adapter parameters. The full Fun-Control backbone and its learned control conditioning remain required.

The source training bundle SHA256 is 47f94f7fc85ed53b5480b28c2bfa3a05659bcf4aeb1278004c0860224787e78a. Both exported formats were compared tensor-by-tensor to its generator_lora state. All 3,600 LoRA video inference receipts from this evaluation record the same source checksum, complete loading of 600 tensors / 300 target layers, four denoising steps and CFG 1.

Loading

Use the native VideoX-Fun Wan2.2-Fun-5B-Control pipeline. The evaluated VideoX-Fun source revision was 1fd9ed9208ba280f6b48c542a7739e636f4e2a14. Start from its 5B Control example.

Download the adapter and loader:

from huggingface_hub import snapshot_download

release = snapshot_download(
    "Perflow-Shuai/WAN-Fun-LoRA-5B-Control-TrainCFG4-4Step-CFG1-step1600",
    allow_patterns=["generator_lora.pt", "perflow_lora.py", "load_wan_fun_lora.py"],
)

After the example loads the full Control transformer, and before applying CPU/group offloading, attach this adapter once:

import sys
from pathlib import Path

sys.path.insert(0, release)
from load_wan_fun_lora import load_wan_fun_lora

receipt = load_wan_fun_lora(
    transformer,
    Path(release) / "generator_lora.pt",
    strength=1.0,
)
assert receipt["loaded_targets"] == 300
assert receipt["loaded_tensors"] == 600

In that example, set sampler_name="Flow_Unipc", num_inference_steps=4, guidance_scale=1.0, shift=5, and enable_teacache=False. Its UniPC branch sets the scheduler constructor's shift to 1 and the pipeline call applies runtime shift 5. Supply your prompt and dynamic depth video via its existing control-video inputs. Preserve the model's depth/inpainting/reference conditioning preprocessing, including the learned ref_conv behavior.

adapter_model.safetensors plus adapter_config.json is the equivalent PEFT/native-Wan export. Keys use blocks.*.self_attn.*, cross_attn.*, and ffn.*. Diffusers-specific key conversion and generic ComfyUI LoRA loading were not validated for this release. The included loader takes generator_lora.pt; its shared implementation is in perflow_lora.py.

PAI-Bench-C Depth evaluation

We evaluated 600 Depth cases, with six prompt variants per control for diversity (3,600 generated videos per local model). Local outputs use 121 frames at 24 FPS and seed 43. The four-step variants use identical inputs, resolution, seed and scheduler; their difference is this adapter at strength 1.

The official row is quoted from PAI-Bench and was not run by us. Local main metrics use its contemporaneous legacy evaluator; diversity uses the current implementation, whose source is identical across the two snapshots. Exact revisions and values are in evaluation.json. The official run's private configuration is unavailable, so the local multistep row is a best-effort reproduction. Legacy depth RMSE has outliers; these rows should not be read as proof that the LoRA surpasses the official base overall.

Compared with direct four-step inference, this LoRA improves Depth, Quality, Diversity and slightly Edge F1, while Blur SSIM and Mask mIoU decrease. Blur SSIM measures similarity after blurring and is not a sharpness metric. The adapter was trained for Depth; other control modalities and transfer to unmodified Wan backbones are not established by these results.

Files and terms

• generator_lora.pt: generator-only payload with generator_lora and step.
• adapter_model.safetensors, adapter_config.json: equivalent FP32 adapter.
• load_wan_fun_lora.py, perflow_lora.py: strict native-Wan loader.
• training_config.yaml, inference_config.yaml: training and inference settings.
• release_metadata.json, provenance.json, evaluation.json, SHA256SUMS: provenance, measured results and checksums.

The upstream Fun-Control model card declares Apache-2.0; the linked depth dataset declares MIT and the caption dataset declares Apache-2.0. This release follows the preceding depth-conditioned adapter's license: other designation because upstream source-video rights are not established by those dataset labels alone. Consult the respective upstream terms. Original training videos, captions, base-model weights and critic weights are not redistributed here.