Echo-WM

Echo-WM

Open and Enterable Omnimodal World Models

📄 Paper · 🌐 Project · 💻 Code

Echo-WM is an omnimodal world model for enterable generative media. Given a reference image, a structured world description, and a navigation sequence, one model jointly generates 720p video, environmental sound, music, and speech, and keeps going across first-person and third-person viewpoints.

It organizes interaction around camera intent: discrete WASD / IJKL commands and continuous 6-DoF poses share one metric-scale relative trajectory. First-person scenes become observer motion; third-person scenes learn camera–character coupling from data, without a view-specific controller.

The current public weights are built on LTX-2.3. See the JoyAI-Echo echo_wm/ package for install, Gradio, and the Action DSL.

Academic research and non-commercial use only. The project remains under the LTX-2 Community License.

Checkpoints

Both files are merged checkpoints. No extra action adapter is required.

You still need the Gemma 3 text encoder (gated):

hf download Echo-Team/Echo-WM --local-dir checkpoints
hf download google/gemma-3-12b-it-qat-q4_0-unquantized --local-dir checkpoints/gemma-3

Use the unquantized Gemma weights. The encoder runs in bfloat16.

Quickstart

git clone https://github.com/jd-opensource/JoyAI-Echo.git
cd JoyAI-Echo/echo_wm
conda create -n echo-wm python=3.11 -y && conda activate echo-wm
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Base (~10 s, six-field prompt):

python scripts/run_wm_case.py \
  --case examples/wm_cases/0010 \
  --checkpoint checkpoints/echo-wm-base.safetensors \
  --gemma-path checkpoints/gemma-3 \
  --output-dir outputs/wm_cases

Flash Preview (4-step causal):

python scripts/run_wm_case_causal.py \
  --case examples/wm_causal_cases/0079 \
  --checkpoint checkpoints/echo-wm-flash.safetensors \
  --gemma-path checkpoints/gemma-3 \
  --video_local_attn_size 19 \
  --video_sink_size 7 \
  --video_chunk_size 3 \
  --output-dir outputs/wm_cases_causal

Full CLI, Gradio, and multi-GPU scripts: echo_wm/README.md and echo_wm/README_CAUSAL.md.

Control

Camera intent is a relative 6-DoF trajectory. At inference, keyboard segments key-frames are joined by commas:

w/s    forward / backward
a/d    strafe left / right
i/k    pitch up / down
j/l    yaw left / right
none   hold still

Example: w-60,a-60,w-60,d-60. Keys can be combined (wj-60). Base prompts use the six fields in PROMPT_SKILL.md: Environment, Character, Style, Perspective, Sounds, Speech. Flash Preview cases keep the original WBench captions.

Results

On the 158-case WBench Navigation split, Echo-WM ranks first by average. Echo-WM Flash keeps most of that score after 4-step causal distillation, and leads Interaction.

On SANA-WM-Bench (241-frame), Echo-WM reports the highest VBench Overall on both Simple (83.91) and Hard (83.96) splits. Numbers and protocol: arXiv:2608.23189.

Method (short)

1. Unified camera intent — discrete commands and metric poses map to one dataset-calibrated relative 6-DoF trajectory; relative UCPE injects geometry into video attention.
2. World data engine — internal gameplay, human-played gameplay, UE simulation, and general web video, mixed as AV-rich / control-clean / balanced sets.
3. Progressive training — AV-CPT → Action-SFT (frozen backbone, camera branch only) → Joint-FT → causal / SGF post-training for Flash.
4. Streaming cache — Flash uses a sink+FIFO KV window (video / audio / A2V / V2A / UCPE). Long-horizon Flash is not in this drop.

Roadmap

Echo-WM is on LTX-2.3 today. Next: LTX-2.5 Base / Causal, then a tighter accel stack.

Backbone

• LTX-2.3 · Base — bidirectional audio-visual DiT used by Echo-WM Base (~10 s).
• LTX-2.3 · Flash / Causal — chunk-causal attention, KV-cache rollout, 4-step Flash Preview.
• LTX-2.5 · Base — load official LTX-2.5 weights (Gemma 4 TE, 2.5 VAE / DiT) into the bidirectional path.
• LTX-2.5 · Causal — the same Flash recipe on 2.5: block-causal masks, sink+FIFO cache, few-step student.

Accel

• Sparse attention — SageAttention and similar sparse / low-bit kernels on video, audio, and UCPE branches.
• FlashAttention / FlashInfer — fused attention for long causal windows without blowing up HBM.
• Paged KV-cache — variable-length cache so rollouts stay bounded; rebase RoPE and UCPE when tokens evict.
• FP8 / TensorRT — compile the DiT forward at lower precision for decode-time throughput.

Citation

@article{zhang2026echowm,
  title         = {EchoWM: Open and Enterable Omnimodal World Models},
  author        = {Zhang, Songchun and Li, Yaowei and Zhuang, Junhao and Jin, Weiyang and Wang, Haoyu and Lu, Xin and Sun, Yilang and Zhang, Shiyi and Li, Haoran and Ma, Xiaoxiao and Li, Yuming and Liu, Yijun and Su, Yaofeng and Ma, Yanwen and Wu, Haoyu and Su, Zihan and Ma, Yue and Zhang, Lvmin and Huang, Haoyang and Xue, Zeyue and Rao, Anyi and Duan, Nan},
  journal       = {arXiv preprint arXiv:2608.23189},
  year          = {2026},
  eprint        = {2608.23189},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2608.23189}
}

License

This project is based on LTX-2 by Lightricks Ltd. Portions were modified by JD.com for academic and research use only. For commercial use of LTX-2 or its derivatives, contact Lightricks Ltd. See JoyAI-Echo LICENSE.