Motius-Bailando-AISTPP

<h1 align="center">Bailando Model Card</h1> <p align="center"> <strong>Actor-critic music-to-dance generation with a learned choreographic memory.</strong> </p> <p align="center"> <a href="https://arxiv.org/abs/2203.13055">Paper</a> | <a href="https://www.mmlab-ntu.com/project/bailando/">Project Page</a> | <a href="https://github.com/lisiyao21/Bailando">Original GitHub</a> | <a href="https://huggingface.co/ZeyuLing/Motius-Bailando-AISTPP">Motius Checkpoint</a> | <a href="https://github.com/ZeyuLing/Motius/blob/main/docs/tasks/music_to_dance.md">Task Protocol</a> </p>

Bailando is the CVPR 2022 oral work Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory. It learns upper- and lower-body VQ codebooks, then composes those dance units autoregressively from music. The Motius release implements the model, audio frontend, AIST++ dataset, pipeline, representation bridge, and evaluator without importing an external checkout at runtime.

Preview

<table> <tr> <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_break_gBR_mBR0_smpl_mesh_512_30fps.gif" alt="Bailando break dance"></td> <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_krump_gKR_mKR2_smpl_mesh_512_30fps.gif" alt="Bailando krump dance"></td> <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_waacking_gWA_mWA0_smpl_mesh_512_30fps.gif" alt="Bailando waacking dance"></td> </tr> <tr> <td align="center">_{Break / <code>gBR...mBR0</code>}</td> <td align="center">_{Krump / <code>gKR...mKR2</code>}</td> <td align="center">_{Waacking / <code>gWA...mWA0</code>}</td> </tr> </table>

The previews are distinct AIST++ evaluation outputs rendered as neutral SMPL meshes at 512x512 and 30 fps. Their position-IK fit errors are 13.75, 13.89, and 13.71 mm. MP4 sources and fit reports are stored beside the GIF assets.

Release Snapshot

The Hugging Face artifact contains both VQ-VAE branches and the actor-critic GPT as safetensors, plus the complete architecture config, source hashes, license, and attribution. It does not require an upstream repository or a second checkpoint download.

Usage

Install the music frontend dependencies:

python -m pip install -e '.[music-to-dance]'

Generate from an audio file:

from motius.pipelines.bailando import BailandoPipeline

pipe = BailandoPipeline.from_pretrained(
    "ZeyuLing/Motius-Bailando-AISTPP",
    device="cuda",
)
result = pipe("music.wav")

print(result.joints.shape)       # (batch, frames, 24, 3)
print(result.music_features.shape)  # (batch, music_frames, 438)

For exact benchmark reproduction, pass the released 438D AIST++ feature stream and paired initial motion. Only the first upper/lower VQ token initializes the generation, matching the official script:

result = pipe(
    music_features=music_features_7p5fps,
    initial_motion=paired_gt_smpl24,
)

Without initial_motion, the public demo seed (423, 12) is used. Raw-audio inference is a convenience path; use released precomputed features when exact paper parity across audio-library versions matters.

Evaluation

Motius ran the converted official epoch-500 VQ-VAE and epoch-10 GPT on all 40 cross-modal validation/test cases. FID and diversity use the 1,320 valid motion PKLs in the AIST++ v1 reference archive. Generated features use the first 1,200 frames; reference features use complete sequences; BeatAlign uses the complete generated sequence and the paired 60 fps music-beat stream.

Lower is better for FID, higher is better for BeatAlign, and diversity is interpreted relative to GT. The paper values are shown as parity targets and are not copied into the reproduced row.

Physical Diagnostics

These Motius joint-level diagnostics use the common SMPL-22 subset. They are not metrics from the Bailando paper.

Dynamic is an expressiveness statistic to compare with GT rather than minimize. The floor-dependent diagnostics are reported in native metric units.

Motion Representation

The public representation name is aistpp_smpl24_joints, shape (T,24,3) in metres with Y up. Joints 0:22 are the standard SMPL body chain and convert exactly to smpl22_joints:

from motius.motion import convert_motion

smpl22 = convert_motion(
    result.joints[0],
    source="aistpp_smpl24_joints",
    target="smpl22_joints",
)

Conversion to motion135 and SMPL mesh uses position IK because the generated tensor stores joint positions rather than local rotations. The three preview reports expose the resulting fit errors instead of hiding this lossy step.

Reproduction Audit

The model code remains under the upstream S-Lab License 1.0. AIST++ annotations are CC BY 4.0. See the artifact LICENSE and ATTRIBUTIONS.md before redistribution or commercial use.

Citation

@inproceedings{siyao2022bailando,
  title={Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory},
  author={Siyao, Li and Yu, Weijiang and Gu, Tianpei and Lin, Chunze and Wang, Quan and Qian, Chen and Loy, Chen Change and Liu, Ziwei},
  booktitle={CVPR},
  year={2022}
}

Direct Loading

from motius import Pipeline

pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-Bailando-AISTPP")