VIPRA-reproduce

VIPER reproduction

From-scratch reproduction of VIPER: Visual In-Context Physics Reasoning for Physically Plausible Video Generation (arXiv 2607.23472v1, tech report, no official code release).

📖 Main documentation is in Chinese: README_zh.md — it explains what the paper does, how the method works, and what every pipeline step means.

📊 Results: RESULTS_zh.md — dataset statistics, throughput, and the engineering issues that had to be fixed to make it run.

What this is

VIPER conditions an image-to-video generator on a reference video that demonstrates a physical process, so the physics transfers to a new target scene. A frozen MLLM reads the reference alongside learnable query tokens; their hidden states become physics condition tokens that are concatenated with the text context of a Wan2.2 DiT. Trained in three hierarchical stages.

viper/          model, data pipeline, training, inference, evaluation
scripts/        slurm / torchrun launchers
third_party/    official Wan2.2 code (DiT / VAE / umT5 definitions)

Quick start

source env.sh          # all caches point at lustre; nothing is written to $HOME
sbatch scripts/full_run.sbatch

Components

---

Output paths — what everything is

👀 Start here: results/comparison/<ref_id>__<tgt_id>/

Nine held-out validation cases. Each folder contains the full picture:

baseline.mp4 and viper.mp4 use identical seeds, so any difference between them is attributable to the reference stream alone.

Metrics

Our VIPER-19K rebuild

Built from the full csusupergear/WISA-80K-wan480p-16fps-81f subset (276 shards, 27543 clips) via scripts/fetch_wisa.py + scripts/scaleup_{1,2,3}_*.sbatch. The pilot column is the earlier 20-shard run, kept for reference.

What the filters cut, at full scale: 14762 near-static (motion_score < 1.0), 4126 non-mechanical labels, 725 low quality, 219 shot transitions, 0 watermarked. The 32B judge kept 934 of ~9600 pairs it got through (9.7%).

Only ~9600 of the 15608 candidates were judged: the judge runs at ~6 s/pair/rank and the 4h interactive limit cuts it off there. --deadline_min makes that stop graceful so the split still runs; raise it on a longer partition to judge the rest.

Logs

logs/ — judge.log (pair filtering), train_stage1_interactive.log (training), comparison.log (video generation), eval_comparison.log, eval_loss.log.

Not published here

Base model weights (models/, ~115GB — fetch from Wan-AI/Wan2.2-I2V-A14B, Qwen/Qwen3-VL-4B-Instruct, Qwen/Qwen3-VL-32B-Instruct), our trained checkpoints, the source video corpus (data/videos/, from csusupergear/WISA-80K-wan480p-16fps-81f), and the precomputed latent cache.

⚠️ Read RESULTS_zh.md before interpreting the videos. Only 150 stage-1 steps were trained (~1% of the paper's 15K budget), so viper.mp4 and baseline.mp4 are near-identical by design — the zero-init connector has not yet grown enough to influence the DiT. These are an infrastructure baseline, not a validation of the method.