helloworld-interactor-lora

helloworld-interactor-lora

⚠️ This LoRA is a fine-tuned Derivative of LTX-2 and is distributed under the LTX-2 Community License (including its use-based restrictions, Attachment A). A complete copy of the license is included in this repository.

Current release: v2 — helloworld-interactor-lora-r32-v2-2500.safetensors (recommended). The original v1 (helloworld-interactor-lora-r32-2000.safetensors, 156-clip corpus) remains for reproducibility. v2 triples the corpus (456 clips — including 300 targeting compound human choreography, subtle low-amplitude gestures, and one-press-one-beat discipline) and trains to 2500 steps. On a 35-pair true-video bench: performs-in-window 62% vs 20% base, human characters 50% vs 6%, visible artifacts 0/35 vs 15/35.

Rank-32 LoRA for Lightricks LTX-2.3 that adds camera-trajectory following from warp-video history conditioning — the self-distillation recipe of "HelloWorld: Enabling Socially Interactive Characters in Video World Models" (Ouyang, Liu, Chu, Zhang, Sato — Alaya Lab / The University of Tokyo), reimplemented independently before the authors' official code release. Paired with the training-free F-press cross-attention mask from the same paper, you get characters that perform at the camera inside a chosen time window while the shot follows a camera path you draw.

• Code, warp pipeline, Eq. 4 mask, training config, reproduction guide: https://github.com/scrappylabsai/helloworld-interactor
• What it is: LoRA rank 32, alpha 32, on all self-attention projection matrices of the video branch (attn1.*), LR 1e-4, flow matching, bf16, on self-generated 1280×704×121 clips with recovered-trajectory warp videos as reference conditioning; camera text excluded from captions (paper §3.2, §5.2). v1: 156 clips / 2000 steps. v2: 456 clips / 2500 steps.
• Training cost: rented RTX PRO 6000s (96 GB), peak 72.3 GB; whole project (v1+v2, all rentals) ~$127.

Results — v2 (full bench, 40 generated characters, base vs LoRA, true-video judge)

Judge: Gemini with true-video input, structured verdicts, blind to which arm is which.

Generalization — the result we didn't train for. On fresh prompts the base model often doesn't even resemble the instruction; with the LoRA the same seed executes it. We believe the adapter learned a grammar (obey the interaction spec, inside the window, hold steady otherwise) rather than content — so it transfers to characters and actions that appear nowhere in training. Public evidence: a wall of 10 brand-new characters (fresh stills, fresh camera warps, unseen interactions — zero overlap with training or bench on any axis): https://interactor-demos.pages.dev/ab4/

Results — v1 (mini-bench, first release)

Qwen3.6-35B-A3B is the paper's judge model; our serving feeds it frame grids rather than raw video, which attenuates absolute scores for both columns (motion between sampled frames is invisible) — the true-video Gemini calibration on the same clips lands at the paper's number within small-n error. Caveats: n=40/n=9 (paper: 400); trained at 121 frames vs the paper's 241; our geometric CamCtrl metric is pending a pose-surveyor bugfix (qualitative camera control — dolly-in, orbit — confirmed side-by-side at fixed seed). Where the trained model missed the directed interaction (2/9 true-video pairs, both subtle low-amplitude gestures — a wink, a sliding object), the untrained baseline missed the same clips: misses degrade to baseline behavior rather than below it, and trained clips showed fewer artifacts (1/9 vs 3/9).

Example pairs (same seed, same prompt, same warp input)

Full-quality mp4s in examples/ — <id>__base.mp4 is stock LTX-2.3, <id>__lora.mp4 is +this LoRA. All characters fully generated.

v2 generalization pairs — characters built from scratch after training, unseen on every axis (still, warp, interaction):

Usage

With the repo's inference driver (LTX-2 packages installed + the repo's vendor patch applied — see the repo's docs/REPRODUCE.md):

python -m helloworld_ltx.infer_warp \
    --distilled-checkpoint-path models/ltx-2.3/ltx-2.3-22b-distilled-1.1.safetensors \
    --gemma-root models/gemma-3-12b \
    --spatial-upsampler-path models/ltx-2.3/ltx-2.3-spatial-upscaler-x2-1.1.safetensors \
    --image inputs/shot/first_frame.png 0 1.0 \
    --warp-npz inputs/shot/warp.npz \
    --scene "A lighthouse keeper on the gallery rail at dusk." \
    --interaction "He turns and waves at the viewer." \
    --quality "Photoreal, natural lighting." \
    --window 24 48 \
    --height 704 --width 1280 --num-frames 121 --frame-rate 24 \
    --lora helloworld-interactor-lora-r32-v2-2500.safetensors 1.0 \
    --seed 42 --output-path out.mp4

--warp-npz comes from the repo's warp/warp.py (choose a trajectory) or warp/annotate.py (recover one from a video). --window is the F-press window in pixel frames at the output rate. Timing control is training-free; this LoRA adds the camera-following.

Choosing LoRA strength. --lora <file> 1.0 is maximum obedience — tightest camera following and window timing. Dialing down to 0.6–0.8 lets more of the base model's ambient life through (background motion, richer prop handling) at the cost of slightly looser camera tracking — in our testing ~0.6–0.7 is the sweet spot for production feel. Same seed comparison: https://interactor-demos.pages.dev/ab4/dial

Dependency license note (Pi3X)

Generating the warp inputs uses the Pi3X / π³ geometry model (arXiv:2507.13347): its weights are CC BY-NC 4.0 (non-commercial) and are NOT included here or in the code repo. The LoRA weights themselves have no Pi3X dependency, but the as-shipped warp pipeline does — commercial users must substitute a permissively-licensed geometry backbone for warp generation.

Credit

All credit for the method to the HelloWorld authors (Alaya Lab / The University of Tokyo) — this is an unofficial reproduction from the paper text, published with full attribution. Watch https://github.com/AlayaLab/HelloWorld for their official release.

Built by ScrappyLabs.

Try it on your own productions — if you direct your own characters with this, we'd love to see the results: open a Discussion on the GitHub repo with your clips and what you directed.