music2dance

Dance2Music — Music2Pose Transformer

Generate dancing skeleton animations from any audio file using a causal transformer trained on AIST++ dance videos.

Author: Emma Genthon · [email protected]
Best checkpoint: epoch 180 (pytorch_model.pt)

What it does

Given a music track, the model autoregressively predicts 33 MediaPipe-style body landmarks at 25 fps, then renders them as a skeleton video synced to the audio.

Audio (32 kHz) → mel + onset + beat features → Music2PoseTransformer → poses (33×3) → skeleton video

Quick start

git clone https://huggingface.co/Emma5099/dance2music
cd dance2music
pip install -r requirements.txt

# Requires ffmpeg on PATH
python generate.py --audio your_song.wav --out dance.mp4

Python API

import soundfile as sf
import librosa
import torch

from audio_features import AUDIO_SR, audio_to_features
from inference import load_checkpoint, generate_poses

waveform, sr = sf.read("song.wav", dtype="float32")
if waveform.ndim > 1:
    waveform = waveform.mean(axis=1)
if sr != AUDIO_SR:
    waveform = librosa.resample(waveform, orig_sr=sr, target_sr=AUDIO_SR)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model, ckpt = load_checkpoint("pytorch_model.pt", device)
poses = generate_poses(
    model, audio_to_features(waveform),
    ckpt["x_mean"], ckpt["x_std"],
    ckpt["y_mean"], ckpt["y_std"],
    device,
)
# poses.shape → (T, 33, 3)

Model

Validation metrics (sample sBT_c01_v7)

The model produces human-shaped skeletons with visible motion. Beat alignment is still limited and improves with more training data and epochs.

Files

Training (optional)

Training uses AIST++ pose sequences paired with audio chunks, weighted losses on position/velocity/acceleration, and scheduled sampling with a rolling 60-frame context. See the full project source for train.py, evaluate.py, and dataset tooling.

Requirements

• Python 3.10+
• PyTorch 2.x
• ffmpeg (for video rendering and audio muxing)
• ~26 MB disk for model weights

License

MIT

Citation

If you use this model, please cite AIST++ and this repository:

@misc{genthon2026dance2music,
  author = {Emma Genthon},
  title = {Dance2Music: Music-Conditioned Pose Generation},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Emma5099/dance2music}}
}