worldrender

Versions

The two versions' inference configs are not interchangeable. --local-attn-size must be each version's own training value: on identical weights and clips, 6 gives 1.62% drift and -1 gives 4.42%.

---

V2

Two additions over V1; the rest of the training setup is unchanged.

Static Sink — logical block 0 is pinned in slot 0 of the ring buffer and re-roped to an out-of-band position (+35), so every block from block 2 onward still attends to the opening block no matter how far the rollout has gone. It comes paired with local_attn_size: 6, which cuts the short-term span from the full rolling history to one previous block; the sink carries the global anchoring that the span gave up.

Reference-image dropout — 25% of training samples zero the reference in feature space (ref_lat and clip_feature), so inference can run with no appearance reference at all.

Files

v2/stage1/step-6000.safetensors        802 tensors = 800 LoRA (r128) + pose_patch_embedding
v2/stage2/step_500_generator_lora.pt   800 tensors, LoRA only (the distilled student)

Loading is the same three-part rule as V1 — base + stage1 + stage2, and the stage2 file alone silently disables the render conditioning.

Inference, with a reference image

python long_infer.py \
  --manifest      <prep>/manifest.jsonl \
  --out-dir       <out> \
  --stage1-ckpt   v2/stage1/step-6000.safetensors \
  --stage2-ckpt   v2/stage2/step_500_generator_lora.pt \
  --config        configs/render_dmd_sink.yaml \
  --context-noise 150 \
  --kv-frames 12 --local-attn-size 6 \
  --decode-chunk 0 --seed 1234

Inference, without a reference image

Add --no-ref. It zeroes ref_lat and clip_feature, matching how the dropout was trained.

Do not feed a black or gray frame instead. Over 8 clips the output brightness tracked the reference's almost exactly, out = 51.0 + 0.483 * ref, r = 0.9868 (black -> 56.8, gray -> 101.2, white -> 180.1, real reference -> 111.1), and all three flat frames collapsed frame std to 52.8-57.1 against 66.9 for a real one. A black frame tells the model the appearance is black; only feature-space zeroing says there is none.

Without a reference the render geometry is still followed exactly — same buildings, same camera path — but texture detail flattens and drift resistance weakens from 1.6% to 11%.

Measured

8 held-out clips, 10 s / 237 frames / 20 blocks, seed 1234:

Medians, not means: V1's mean of 11.00% against a median of 4.53% is entirely two outlier clips (28% and 37%), one of them low-contrast source material. The 4.3% quoted for V1 below is likewise an outlier-excluded figure.

step_500 is an interim checkpoint of a run that is still training (target 3000 steps). It is the newest point that has been fully evaluated, not a converged model.

---

V1

WorldRender — low-poly render → real video

A low-poly geometric render drives block-causal autoregressive generation of the corresponding real video. The render goes into the model's spatial-control slot, so no keypoint or skeleton extraction is involved.

Code: https://github.com/songyiren725/WorldRender

~4.9 fps on one H100. A 30 s rollout (60 blocks) is about as good as a 2 s one (4 blocks):

Jitter is the median of per-frame mean|f[t+1]-f[t]|; drift is the relative change in mean brightness between the first and last eighth. 15x the block count costs 17% more jitter, and drift stops growing after about 6 s.

Files

stage1/step-10000.safetensors         802 tensors = 800 LoRA (r128) + pose_patch_embedding
stage2/step_2500_generator_lora.pt    800 tensors, LoRA only (the distilled student)
teacher/step-8100.safetensors         bidirectional teacher, only needed to redo stage 2

Loading — all three parts are required

base Wan2.1-I2V-14B-720P
  + stage1/step-10000.safetensors        LoRA + pose_patch_embedding
  + stage2/step_2500_generator_lora.pt   distilled LoRA, overlaid on top

The stage2 checkpoint does not contain pose_patch_embedding — PEFT saves only trainable parameters and that module is frozen in stage 2. Load stage2 alone and x = x + pose_patch_embedding(render_latents) becomes the identity: the model silently ignores its only control signal while still producing plausible video. Assert after loading:

w = dict(gen.named_parameters())["model.pose_patch_embedding.weight"]
assert float(w.abs().max()) > 0

The two checkpoints genuinely disagree on key prefix — not a typo:

stage1   saved from the generator wrapper  ->  model.blocks.0...   load into gen
stage2   saved from generator.model        ->        blocks.0...   load into gen.model

pose_patch_embedding, face_adapter, face_encoder and motion_encoder do not exist in the base model, so loading must use strict=False. This corpus has no faces and the face branch is short-circuited throughout — those three modules are randomly initialised and never trained, so any inference path that calls them unconditionally injects random residuals into 8 of the 40 layers.

Inference

python long_infer.py \
  --stage1-ckpt   stage1/step-10000.safetensors \
  --stage2-ckpt   stage2/step_2500_generator_lora.pt \
  --config        configs/render_dmd.yaml \
  --context-noise 150 \
  --kv-frames 12 --local-attn-size -1 \
  --decode-chunk 0 --seed 1234

denoising_step_list = [1000.0, 750.0, 500.0], and it must be float. sinusoidal_embedding_1d ends with return x.to(position.dtype), so an int64 timestep truncates the whole cos/sin embedding (values in [-1,1]) to zero and the output becomes pure noise.

context_noise=150 is the anti-drift lever: noise is added to a generated block before it is written back into the KV cache, which stops the model from becoming over-confident in its own output. Use 0 during training, 150 at inference.

Why step 2500 and not the final 5000

Stage 2 degrades after ~2500 steps. From its own training-time samples, averaged over all 8 ranks:

step:      0     1000   2000   2500   3000   3500   4500   5000
jitter    1.29   3.13   2.26   1.82   3.96   3.67   1.30   2.81
mean       122    119    124    130    140    225    237    157
std         58     61     61     57     55     25     11     48

Rising mean with collapsing std is contrast draining away; by 4500 the samples are nearly white. Re-measured under a corrected long rollout, step_5000 drifts 74% (the picture collapses toward a single tone) against 10.5% for step_2500. Step 2500 is closest to the healthy step-0 baseline on all three numbers.

Not overfitting: 9573 clips yield ~700k distinct windows and 5000 steps cover 5.7% of them. More likely a critic/generator imbalance in the distillation.

Training setup

base            Wan2.1-I2V-14B-720P (single dense DiT, 40 layers — not a MoE)
resolution      480x848 @ 24 fps
stage 1         causal adaptation with teacher forcing, 10000 steps, lr 5e-5 constant
                bidirectional DiT -> block-causal AR generator, 3 latent frames per block
                reference-image KV kept globally visible to every block
stage 2         distribution-matching distillation, 50 -> 3 steps [1000, 750, 500]
                two-pass block replay so a 14B student fits one 8x80GB node
                num_training_frames 12 (4 blocks), lr 1e-5 constant
LoRA            rank 128 on q,k,v,o,ffn.0,ffn.2
data            9573 training clips, 13.6 hours
hardware        8 x H100 80GB, single node, FSDP FULL_SHARD

Known limits

The KV cache holds a bounded window of 12 latent frames, so memory is flat in output length, but the attention span is what training saw and nothing anchors the rollout beyond the reference frame. Drift is therefore not zero — it settles around 4-5% rather than growing without bound.

Clips whose render has low contrast (measured std 40 against 51-58 for the rest) show visibly more jitter: the render reaches the model only through pose_patch_embedding, so low contrast means weak conditioning.

See docs/PITFALLS.md in the repo for the failure modes this code has actually produced, including three diverged training runs, inference emitting pure noise, and a stutter every two seconds from a KV cache wrap bug.