ABot-World-0-5B-LF-Diffusers

ABot-World-0-5B-LF — Modular Diffusers

ABot-World (acvlab/ABot-World-0-5B-LF, Apache-2.0) as a Modular Diffusers pipeline: a real-time, action-conditioned world model finetuned from Wan2.2-TI2V-5B. Starting from an input image and character reference views, it rolls a world out block by block (3 latent frames each), steered by per-block [W, A, S, D, I, J, K, L] keyboard actions.

The transformer is converted to diffusers format (ABotWorldTransformer3DModel); the VAE, text encoder, and tokenizer load from Wan-AI/Wan2.2-TI2V-5B-Diffusers (byte-identical to the files the original checkpoint ships).

Requires diffusers with the ABot-World integration (currently the abot-world-modular branch).

import torch
from diffusers import ModularPipeline
from diffusers.utils import export_to_video, load_image

pipe = ModularPipeline.from_pretrained("YiYiXu/ABot-World-0-5B-LF-Diffusers")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("...")            # the starting frame
refs = [load_image(...) for ...]     # 5 character views: head/left/right/front/back
actions = [[1, 0, 0, 0, 0, 0, 0, 0]] * 8 + [[1, 0, 0, 0, 0, 0, 0, 1]] * 8  # walk, then walk while turning

state = pipe(
    prompt="| unknown |",
    image=image,
    reference_images=refs,
    actions=actions,
    generator=torch.Generator("cpu").manual_seed(0),
)
export_to_video(state.get("videos")[0], "world.mp4", fps=12)

Streaming (a live state after every denoise step and block):

for event in pipe.stream(prompt=..., image=..., reference_images=..., actions=...):
    if event.path == "denoise.rollout":            # one finished block (~1 s of video)
        latents = event.state.get("latents")

For interactive driving, own the loop via the rollout block's loop_step and write new actions into the state between calls.