Motius-UniMuMo

<h1 align="center">UniMuMo Model Card</h1> <p align="center"> <strong>One checkpoint for generation and translation across text, music, and motion.</strong> </p> <p align="center"> <a href="https://arxiv.org/abs/2410.04534">Paper</a> | <a href="https://hanyangclarence.github.io/unimumo_demo/">Project Page</a> | <a href="https://github.com/hanyangclarence/UniMuMo">Original GitHub</a> | <a href="https://huggingface.co/ClarenceY/unimumo">Original Checkpoint</a> | <a href="https://huggingface.co/ZeyuLing/Motius-UniMuMo">Motius Checkpoint</a> | <a href="https://huggingface.co/spaces/ZeyuLing/t2m-humanml3d-leaderboard">T2M Leaderboard</a> | <a href="https://huggingface.co/spaces/ZeyuLing/m2t-humanml3d-leaderboard">M2T Leaderboard</a> | <a href="https://huggingface.co/spaces/ZeyuLing/music-to-dance-aistpp-leaderboard">M2D Leaderboard</a> | <a href="https://huggingface.co/spaces/ZeyuLing/dance-to-music-aistpp-leaderboard">D2M Leaderboard</a> </p>

UniMuMo is the unified text, music, and motion model introduced in UniMuMo: Unified Text, Music and Motion Generation. Motius independently implements its inference architecture and converts the authors' published weights into one self-contained safe artifact. Runtime inference does not import an upstream checkout or download a second text, audio, motion, or caption model.

Preview

• HumanML3D all-case text-to-motion comparison
• HumanML3D all-case motion caption comparison
• Audio-synchronized AIST++ all-case dance comparison
• Motion-synchronized AIST++ all-case music comparison

The T2M page compares UniMuMo's generated SMPL Mesh with every released baseline over all 4,042 selected-caption cases. The M2T page shows the same animated input SMPL Mesh beside every baseline caption for all 4,400 protocol samples. The M2D page contains all 40 AIST++ cases, synchronized audio, native SMPL-24 joints, the fitted SMPL Mesh, orbit, zoom, timeline seeking, and downloadable motion assets. The D2M page contains the official 86 two-second D2M-GAN clips and lets the viewer switch between reference and generated audio while the same input mesh plays.

Release Snapshot

The artifact includes both core shards, Encodec, T5 encoder, T5 captioner, SentencePiece tokenizer, HumanML3D normalization statistics, configuration, and provenance. UniMuMoPipeline.from_pretrained is the only loader needed.

Usage

Install the UniMuMo dependencies and load the complete artifact:

python -m pip install -e '.[unimumo]'

from motius.pipelines.unimumo import UniMuMoPipeline

pipe = UniMuMoPipeline.from_pretrained(
    "ZeyuLing/Motius-UniMuMo",
    device="cuda",
)

Generate synchronized music and motion from two optional text descriptions:

result = pipe.infer_text_to_music_motion(
    music_prompt="an upbeat electronic dance track",
    motion_prompt="a person dances energetically",
    duration_seconds=8.0,
    guidance_scale=4.0,
    seed=7,
)

print(result.waveform.shape, result.sample_rate)  # (256000,), 32000
print(result.motion.shape, result.motion_fps)     # (480, 263), 60.0
print(result.joints.shape)                       # (480, 22, 3)

Use the task-specific routes with the same loaded pipeline:

t2m = pipe.infer_text_to_motion(
    "a person walks in a circle",
    duration_seconds=6.0,
    seed=7,
)
t2music = pipe.infer_text_to_music(
    "a quiet piano melody",
    duration_seconds=6.0,
    seed=7,
)
m2d = pipe.infer_music_to_motion(
    "music.wav",
    motion_prompt="a person performs a street dance",
    guidance_scale=3.0,
    seed=7,
)
motion_music = pipe.infer_motion_to_music(
    t2m.motion,
    input_fps=60.0,
    music_prompt="upbeat percussion",
    seed=7,
)
motion_caption = pipe.infer_motion_to_text(t2m.motion, input_fps=60.0)
music_caption = pipe.infer_music_to_text("music.wav")

Array audio inputs also require sample_rate=.... Motion inputs must be HumanML3D-263; other Motius representations should first be converted through the motion representation API.

Evaluation Results

HumanML3D Text-to-Motion

UniMuMo exposes text-to-motion as a zero-shot route of its joint music-motion generator. Motius evaluates all 4,042 selected-caption protocol cases with one deterministic pass and retrieval groups of 32. HumanML3D Official uses the 4,012 cases for which the released new_joint_vecs reference exists; its retrieval computation uses the largest complete 32-case groups (n=4,000).

HumanML3D and MotionStreamer FID use their native evaluator spaces; uTMR FID uses per-sample L2-normalized embeddings. The weak retrieval scores are reported as measured: this checkpoint supports T2M, but it was not optimized as a dedicated HumanML3D text-to-motion model.

The HumanML3D result above is computed from the codec's native 60 fps output with the phase-aligned [1::3] inverse used by the official UniMuMo data pipeline. A parity audit found and fixed a top-k sampling-order discrepancy; under the upstream dependency versions, motion codes, sampled tokens, and decoded features now match the released implementation exactly. The UniMuMo paper does not report a standalone HumanML3D T2M leaderboard result, so the remaining low retrieval score is recorded as the zero-shot operating point of the released joint model rather than treated as a paper-parity target.

Physical diagnostics over all 4,042 generated SMPL-22 joint sequences:

HumanML3D Motion-to-Text

The shared M2T protocol contains 4,400 official test motions and temporal subclips, three references per sample, semantic retrieval groups of 32, and one deterministic evaluation pass. UniMuMo follows the authors' 10-second captioning protocol: 20 fps HumanML3D input is padded to 200 frames, linearly resampled to 60 fps, encoded, and captioned.

The lexical metrics use the TM2T token/lemma references. BERT raw is the unrescaled RoBERTa-large layer-17 cosine score; the corresponding baseline-rescaled BERTScore is 0.4109. The paper reports R@1=0.520, R@3=0.806, and MM-Dist=2.958, which closely matches this independent run. The leaderboard's raw-reference diagnostic reports BLEU-4 0.1271, ROUGE-L 0.3560, CIDEr 0.3114, and raw BERTScore 0.9075.

AIST++ Music-to-Dance

The common leaderboard evaluates all 40 public cross-modal cases against the complete 1,320-motion AIST++ reference pool. FID_k/FID_g and diversity use the released Bailando 60 fps protocol. uTMR FID uses canonical SMPL-22 joints at 30 fps with per-sample L2-normalized embeddings.

Physical diagnostics on the same generated clips:

For paper parity, a separate first-five-second evaluation gives FID_k=10.7721, FID_g=27.3115, and BeatAlign=0.2430; the paper reports BeatAlign 0.24. The leaderboard uses full generated timelines for every method and does not mix the shorter parity result into rankings.

AIST++ Dance-to-Music

Motion-to-music is a zero-shot route of the same released checkpoint. Motius uses the D2M-GAN list of 86 two-second AIST++ clips, no text prompt, CFG=3, temperature 1.0, top-k 250, and the released onset detector and one-second beat-bin aggregation.

The paper calls the first metric Beats Coverage, but its released evaluator defines it as generated beat bins divided by reference beat bins. It is therefore unbounded: values above 100% mean excess detected beats, not a score better than 100%. Motius displays the less ambiguous name Beat Count Ratio and does not rank it with a higher-is-better arrow.

An earlier 40-case result was invalid and has been removed. The published audio codec stores modern parametrized weight-normalization keys, while the official runtime expects weight_g and weight_v; the permissive loader had silently left convolution weights randomly initialized. The Motius loader now maps the layouts explicitly and rejects every missing, unexpected, or shape-mismatched tensor. After the fix, motion codes, music codes, conditioned motion codes, and decoded waveforms match the official implementation exactly (waveform RMSE and maximum absolute error are both zero).

The public UniMuMo motion archive contains exact HumanML3D-263 features for 13 of the 20 AIST++ source dances in this protocol. The seven filtered dances are reconstructed from the official AIST++ SMPL release using the same public SMPL web rig and HumanML3D preprocessing. The six 119-frame tail clips retain their native length instead of being padded to 120 frames.

Every generated WAV, codec output, input-motion provenance record, seed, and per-case metric is available in the dance_to_music_d2mgan_official86 benchmark folder.

Open the Dance-to-Music leaderboard and synchronized 86-case SMPL/audio viewer.

Motion Representation

The native motion stream is HumanML3D-263. Its root velocities, root height, root-invariant positions, continuous 6D local rotations, local velocities, and foot contacts are normalized with the authors' published statistics and encoded jointly with zero-audio embeddings. The motion codec maps 60 fps motion to the same 50 Hz, four-codebook token clock used by music.

For the AIST++ viewer and evaluator, Motius decodes HumanML3D joints, converts the common SMPL-22 body directly, and extrapolates AIST++ hand joints 22 and 23 from each elbow-to-wrist direction by 0.35x. The official feature evaluator uses 24 joints; uTMR uses only the common SMPL-22 body. SMPL Mesh preview uses position IK because generated joint positions do not uniquely determine axial twist. Across all 40 clips, the fitted mesh has 28.16 mm mean joint MPJPE.

Reproduction Audit

The audited upstream source and checkpoint declare no license. Motius records that fact rather than inferring redistribution terms. Users remain responsible for obtaining permission appropriate to their use case.

Citation

@article{yang2024unimumo,
  title={UniMuMo: Unified Text, Music and Motion Generation},
  author={Yang, Han and Su, Kun and Zhang, Yutong and Chen, Jiaben and Qian, Kaizhi and Liu, Gaowen and Gan, Chuang},
  journal={arXiv preprint arXiv:2410.04534},
  year={2024}
}

Direct Loading

from motius import Pipeline

pipeline = Pipeline.from_pretrained("ZeyuLing/Motius-UniMuMo")