many-views-one-world

SyncWM: Inference Release

What this release is

This is the anonymous ICLR 2027 SyncWM artifact for mixed-expert video inference. It packages the public inference implementation and two step-8500 delta checkpoints. It does not include the LingBot-World base model, the VAE, or dataset-derived condition caches.

The checkpoints are adapters, not standalone generative models. Both require the external base model and inputs listed below.

Included files/checkpoints

Mixed LOW/HIGH route

The sampler uses both experts during denoising:

HIGH non-interaction adapter
        t >= 0.947
             |
      boundary sigma 0.947
             |
         t < 0.947
LOW interaction-enabled adapter

Interaction conditioning is active only on LOW. See docs/INFERENCE.md for the dense-resolution contract, interaction projection, condition fusion, and routing details.

External dependencies

Install a compatible Python/PyTorch environment using the packaged dependency profiles. The provided CUDA profile targets CUDA 12.4; the installed PyTorch build must also be compatible with the host NVIDIA driver.

Quick start

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements/cuda124.txt
python -m pip install -r requirements/base.txt
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation

Run from the release root after preparing the external dependencies:

PYTHONPATH="$PWD/tools:$PWD/vendor/lingbot" \
python tools/sample_memory_dense_adapter_interaction_eval_v1.py \
  --lingbot-repo vendor/lingbot \
  --ckpt-dir /path/to/lingbot-world-base-cam \
  --adapter-checkpoint-low checkpoints/low_interaction_step8500.pt \
  --adapter-checkpoint-high checkpoints/high_adapter_step8500.pt \
  --map-manifest /path/to/dense_map_manifest.json \
  --cache-manifest /path/to/aligned_cache_manifest.jsonl \
  --state-cache-manifest /path/to/state_interaction_manifest.jsonl \
  --clip-root /path/to/clips \
  --out-dir outputs/mixed_route \
  --split test \
  --clip-id EXAMPLE_CLIP_ID \
  --interaction-variant true \
  --boundary-override 0.947 \
  --latent-frames 21 \
  --chunk-size 3 \
  --base-sampling-steps 70 \
  --base-guide-scale 5.0 \
  --base-shift 3.0 \
  --target-height 480 \
  --target-width 832 \
  --seed 20260801 \
  --no-offload-model

This configuration requests an 81-frame, 832x480 video at 16 FPS. The shown seed is the sampler default; use the evaluation-specific seed when reproducing a particular reported result.

Checkpoints

Both files are sanitized inference-only PyTorch checkpoints. Optimizer state, training paths, run directories, and resume metadata are excluded. All retained tensor keys, shapes, dtypes, and values were checked against their source training checkpoints.

Reproducibility notes

• The packaged code and checkpoint structures have passed static validation, including CLI parsing, state-group compatibility, tensor inventories, and checkpoint hashes.
• A complete mixed-route generation has not been rerun from this packaged release under the currently available hardware conditions.
• LOW uses a [7,240,416] runtime dense condition. HIGH was trained with [7,176,320], but at runtime it consumes the same shared [7,240,416] manifest as LOW. This mixed-resolution lineage is a known validation caveat.
• Exact public revisions for the external base model, VAE, and text-model assets remain to be pinned before claiming a fully self-contained artifact.

Detailed docs

• docs/INFERENCE.md: routing, input contracts, condition fusion, runtime dense resolution, base-model layout, and outputs.
• docs/CHECKPOINTS.md: checkpoint structure and state groups.
• checkpoint_manifest.json: exact checkpoint sizes, hashes, tensor names, shapes, dtypes, and parameter counts.
• LICENSE and THIRD_PARTY_NOTICES.md: project and third-party licensing information.