Motius-EDGE-AISTPP

<h1 align="center">EDGE Model Card</h1> <p align="center"> <strong>Editable diffusion-based dance generation from music.</strong> </p> <p align="center"> <a href="https://openaccess.thecvf.com/content/CVPR2023/papers/Tseng_EDGE_Editable_Dance_Generation_From_Music_CVPR_2023_paper.pdf">Paper</a> | <a href="https://edge-dance.github.io/">Project Page</a> | <a href="https://github.com/Stanford-TML/EDGE">Original GitHub</a> | <a href="https://huggingface.co/ZeyuLing/Motius-EDGE-AISTPP">Motius Checkpoint</a> | <a href="https://zeyuling-music-to-dance-aistpp-leaderboard.static.hf.space/cases/index.html">M2D Leaderboard + Preview</a> </p>

EDGE is the CVPR 2023 work EDGE: Editable Dance Generation From Music. Motius reproduces the released AIST++ network, cosine DDIM sampler, classifier- free guidance schedule, long-sequence overlap, Jukebox conditioning contract, motion decoder, and coordinate conversion without importing an upstream checkout at runtime.

Preview

Open the unified audio-synchronized 40-case M2D comparison. Every case shows GT, Bailando, and EDGE in one comparison page. The EDGE scene overlays its native SMPL-24 skeleton on the SMPL mesh decoded from the same local rotations. It supports orbit, zoom, timeline seeking, audio synchronization, NPZ export, and FBX export.

The viewer preserves the generated heading and XZ trajectory. It applies one clip-wide vertical translation for display and never grounds individual frames, so jumps, foot slide, and root-height drift remain visible.

Release Snapshot

Usage

Install Motius and the official Jukebox feature frontend:

python -m pip install -e '.[music-to-dance]'
python -m pip install 'jukemirlib @ git+https://github.com/rodrigo-castellon/jukemirlib.git'

Generate from raw audio:

from motius.pipelines.edge import EDGEPipeline

pipe = EDGEPipeline.from_pretrained(
    "ZeyuLing/Motius-EDGE-AISTPP",
    device="cuda",
)
result = pipe(
    "music.wav",
    seed=7,
    jukebox_cache_dir="checkpoints/models/edge/jukebox_cache",
)

print(result.joints.shape)       # (frames, 24, 3), Y-up metres
print(result.edge_motion.shape)  # (frames, 151)
print(result.motion_135.shape)    # (frames, 135)

The released EDGE checkpoint does not contain the frozen OpenAI Jukebox 5B frontend. jukemirlib downloads its VQ-VAE and level-2 prior on first use. For reproducible offline setup, the expected files are:

Precomputed (N,150,4800) feature windows can be passed directly and avoid loading Jukebox:

result = pipe(music_features=features, seed=7)

Reproduce the public 40-case package with resumable per-case outputs. Extract the ten unique music tracks once:

python tools/infer_edge_aistpp.py \
  --checkpoint ZeyuLing/Motius-EDGE-AISTPP \
  --case-manifest docs/leaderboards/hf_space_music_to_dance/cases/manifest.json \
  --audio-root docs/leaderboards/hf_space_music_to_dance/cases \
  --feature-root outputs/edge/aistpp_jukebox_features \
  --output outputs/edge/aistpp_official_40 \
  --extract-features-only \
  --jukebox-cache-dir checkpoints/models/edge/jukebox_cache

Then generate all 40 cases:

python tools/infer_edge_aistpp.py \
  --checkpoint ZeyuLing/Motius-EDGE-AISTPP \
  --case-manifest docs/leaderboards/hf_space_music_to_dance/cases/manifest.json \
  --feature-root outputs/edge/aistpp_jukebox_features \
  --output outputs/edge/aistpp_official_40 \
  --seed 20260721

The manifest mode uses each case's exact duration, assigns a unique deterministic seed, and preserves EDGE's native 30 fps timeline. Jukebox features are shared by music id, so the four cases associated with one song do not recompute the 10.3 GB frontend.

Evaluation

Motius evaluates the released checkpoint on the same 40 AIST++ cross-modal cases used by the public Bailando row. Official kinetic/geometric features are computed after phase-aligned interpolation from EDGE's native 30 fps output to the Bailando 60 fps feature timeline. BeatAlign uses the native 30 fps timing; uTMR uses canonical 30 fps SMPL-22 joints and per-sample L2-normalized embeddings against the fixed 1,320-motion reference pool.

Lower is better for FID, higher is better for BeatAlign, and diversity should be interpreted relative to GT. These are Motius common-protocol measurements, not values copied from the EDGE paper. EDGE's paper evaluates its own generated sample population and introduces PFC; its published protocol is not presented as if it were the same 40-case Bailando benchmark.

Physical Diagnostics

These diagnostics use the shared SMPL-22 subset on the common 60 fps metric timeline. Dynamic measures expressiveness relative to GT rather than an error to minimize.

Motion Representation

EDGE-151 is [contacts(4), root_position(3), SMPL24_local_rot6d(144)]. EDGE stores PyTorch3D's first-two-rows 6D rotation convention in a Z-up frame. Motius decodes native joints exactly, then converts to Y-up motion135 by:

1. applying the inverse 90-degree X basis transform to root position and root rotation;
2. preserving the other local rotations;
3. re-encoding rotations in Motius's motion135 first-two-columns convention.

This route uses no IK. Mesh shape can still differ slightly from the released fixed joint offsets when a different SMPL gender or beta is selected.

Verification

Citation

@inproceedings{tseng2023edge,
  title={EDGE: Editable Dance Generation From Music},
  author={Tseng, Jonathan and Castellon, Rodrigo and Liu, C. Karen},
  booktitle={CVPR},
  year={2023}
}

Direct Loading

from motius import Pipeline

pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-EDGE-AISTPP")