video-to-animation-research

Video-to-Animation Research

Research repo archiving the attempts, data and findings from converting monocular videos of dancing people into bone animations driving a Ready Player Me / Mixamo avatar. Parked in April 2026; archived for future resume.

Goal

Given any video of a person dancing, produce a playable .glb animation on an RPM-style avatar (e.g. Sofia, Avaturn, Rayan) that visually matches the movement in the video. Quality is measured by a MediaPipe-based pose similarity score (MP score, 0–100).

Target: MP ≥ 60. Best achieved: MP ≈ 39.3 (wham-v18). Ceiling appears to be driven by the extractor's own accuracy on dance motion, not by the retargeting stage.

Pipeline overview

video.mp4
  │
  ├─► [extractor] ──► pose sequence (SMPL, 3D joints, or rotations)
  │                     ├─ WHAM / SMPL-est-X / 4D-Humans / PromptHMR-Vid
  │                     ├─ GVHMR / TRAM / HybrIK
  │                     └─ MediaPipe 3D
  │
  ├─► [retarget] ────► bone rotations on Mixamo skeleton
  │                     ├─ ARP (Auto-Rig Pro) — most used
  │                     ├─ analytical IK from 3D joints
  │                     └─ MoCapAnything-style Procrustes+swing
  │
  ├─► [post] ────────► smoothing / correction
  │                     ├─ SmoothNet
  │                     ├─ DNO / MotionFix refinement (not tried)
  │                     └─ learned-correction (static Δ per bone)
  │
  └─► [avatar bake] ─► final .glb with avatar mesh + animation

Alternative (Option C, MVP implemented): skip extraction entirely and retrieve the closest match from a pre-made Mixamo library via MediaPipe keypoints + DTW — scripts/dtw_match.py.

Approaches tried (chronological)

Key findings (the ones to remember)

1. Closure test failed (GT animation → our pipeline → compare to GT = MP 17.4). Any extraction approach is capped at ~60 MP because the pipeline itself drops ~60 points via avatar bind-pose, axis conventions, bone-roll mismatches.

2. Bind-pose mismatch dominates error. SMPL T-pose vs Sofia A-pose = 54.7° delta at the shoulder. Per-bone error breakdown from bone_error_metric.py:

   • Hips: 170° (camera-frame/world-frame conversion)
   • UpLegs: 170° (same)
   • Shoulders: 125° (T-pose vs A-pose)
   • Arms: 84°
   • Spine: 10°
   • Hands: 26°

3. "Fix the bind-pose" ideas that did NOT work:

   • Re-rig Sofia as SMPL (manual Blender) — mesh distorted even with shape-key clean-up
   • Swap to known T-pose avatar (avaturn/rayan) — didn't materially change score
   • Learned static correction per bone (qlog mean) — error is frame-dependent, not static

4. MoCapAnything ≠ free lunch without their model. The paper's real contribution is a trained 3D joint predictor (DINOv2 + 4D mesh). Their §3.5 IK recipe applied to WHAM-derived joints cannot exceed WHAM's ceiling, because those joints come from WHAM's own SMPL forward-kinematics. Code not released as of April 2026.

5. WHAM output convention (critical, repeatedly rediscovered):

   • poses_body = 23 SMPL-relative rotmats (body joints 1..23)
   • poses_root_world = root rotation in world frame
   • trans_world = root translation
   • verts_cam = 6890 SMPL vertices in camera space
   • Default Y-down unless return_y_up=True. yup2ydown flips Y-axis on root rotation and translation only — body-relative rots are frame-independent.
   • For Mixamo retargeting the fast path is poses_body + poses_root_world directly, not via joint positions.

6. Pipeline loses information even for the "identity" case (GT → extract → retarget → render → score vs GT = 17.4). Sources: (a) GLB animation sampling at 30fps, (b) armature empty-node vs pose-bone copy conversions, (c) rotation order / quaternion sign ambiguities.

7. HumanMM repo is empty (README only, no code). No forks/mirrors exist. PromptHMR-Vid already integrates TRAM and tests similarly to WHAM.

Approaches worth trying next

Priority order for future-me:

1. MotionFix / DNO post-refinement over wham-v18 output. Diffusion-based motion denoising. Expected +5–10 MP. Low risk, well-documented.
2. Physics-based IK (joint limits + contact constraints). ECCV'24 "Biomechanically Accurate Neural IK". Useful if extraction is decent but poses look unnatural.
3. Trained retarget model. Take our 5–6 annotated GT videos + WHAM extractions → train a neural net that maps SMPL pose → Mixamo-correct pose per bone. Feedback loop with Groq/LLaMA already memoised (project_groq_autooptimize.md).
4. Better extractor. Wait for HumanMM release (CVPR 2025 specialist on multi-shot dance); or test GVHMR (script exists: scripts/gvhmr_client.py); or PromptHMR-Vid tuning.
5. Retrieval (Option C2). MVP in this repo is C1 (one-shot nearest-neighbour). C2 adds a sliding DTW window with crossfades between clips — likely produces visually much better output than any single extractor on dances.

Directory layout

Key scripts

Extraction:

• wham_to_joints3d.py — convert WHAM pkl → per-frame 3D joint JSON.
• fourdh_to_json.py — same for 4D-Humans.
• hybrik_client.py, gvhmr_client.py — HTTP clients for remote extraction services.
• extract_video_signature.py — MediaPipe → 12-joint body-local npz.
• extract_library_signatures.py — Blender batch extractor for Mixamo library GLBs.

Retargeting:

• arp_retarget.py — Auto-Rig Pro retargeting (champion path). Supports --mirror-target-rest for closure tests.
• mocapanything_ik.py — Procrustes + swing IK from 3D joints. §3.5 of arxiv 2512.10881.
• joints3d_to_animation.py — shortest-arc IK (failed, score 0).
• smpl_to_rpm_retarget.py, sofia_source_retarget.py, retarget_fk.py, rokoko_retarget.py, blender_retarget.py — alternative retargeters.
• apply_hybrik_correction.py, hybrik_calibration.py — HybrIK-specific post.

Diagnostics:

• bone_error_metric.py — per-bone quaternion angular error GT vs extracted.
• closure_test.py — GT → our JSON format → compare; validates pipeline correctness.
• learn_correction.py — attempt to learn static per-bone Δ (failed, 30.5° residual).
• compare_extraction.py, compare_mocap.py, compare_rotations.py — various side-by-side comparators.

Retrieval (Option C, MVP):

• dtw_match.py — DTW matching with motion-energy penalty (fix for idle-bias). Results:
  • 01-idle → M_Standing_Idle_002 (cost 0.25, correct cluster)
  • 02-dance1 → F_Dances_006 (cost 0.80, GT was F_Dances_001)
  • 03-dance2 → M_Dances_003 (cost 0.75, GT was F_Dances_005)

Rendering:

• render_animated_avatar.py — HD render avatar + animation via workbench engine.
• render_wham_mesh.py, render_smpl_verts.py — render raw extractor output (no retarget).

Running experiments (reproduction)

Pre-req: Blender ≥ 4.5, Python ≥ 3.10, pip install mediapipe opencv-python numpy joblib.

# 1. Extract library signatures (one-time, ~3 min for 121 GLBs)
blender -b -P scripts/extract_library_signatures.py -- \
  --lib-dir <path-to-rpm-library> --out motion-library/library.npz

# 2. Extract video signature
python3 scripts/extract_video_signature.py \
  --video videos/02-dance1.mp4 \
  --out motion-library/videos/02-dance1.npz \
  --model <path-to>/pose_landmarker_heavy.task

# 3. Match
python3 scripts/dtw_match.py \
  --library motion-library/library.npz \
  --video motion-library/videos/02-dance1.npz \
  --subsample 3 --top-k 10 --mode dtw

# 4. MoCapAnything IK from WHAM joints (alternative path)
blender -b -P scripts/mocapanything_ik.py -- \
  --avatar <path-to>/avaturn.glb \
  --joints extractions/wham-joints3d/02-dance1-joints3d.json \
  --out mocapik/02-dance1-wham-mocapik-arp.glb

Related work / external deps (not stored here, reconstruct if needed)

Originally under tests/ml-mocap/{extern,humor,synsp,video2anim,data}/, excluded to keep repo size down. Fetch from:

• WHAM: https://github.com/yohanshin/WHAM
• 4D-Humans / HMR2.0: https://github.com/shubham-goel/4D-Humans
• PromptHMR / PromptHMR-Vid: https://github.com/rawalkhirodkar/prompthmr
• GVHMR: https://github.com/zju3dv/GVHMR
• SMPL-est-X: https://github.com/SMPLCap/SMPLest-X
• HybrIK: https://github.com/jeffffffli/HybrIK
• MoCapAnything: https://animotionlab.github.io/MoCapAnything/ (code not released, Apr 2026)
• HumanMM: CVPR 2025 — repo README-only as of Apr 2026
• Auto-Rig Pro (Blender addon): paid, https://blendermarket.com/products/auto-rig-pro

Models required locally:

• pose_landmarker_heavy.task (MediaPipe) — https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_heavy/float16/latest/pose_landmarker_heavy.task
• SMPL body model (basicModel_neutral_lbs_10_207_0_v1.0.0.pkl) — https://smpl.is.tue.mpg.de/
• SMPL-X models — https://smpl-x.is.tue.mpg.de/

License

Internal research. MIT on original code in scripts/. External test videos and Mixamo-derived GLBs retain their original licenses.

Status

Archived April 2026. Ceiling MP ≈ 39 was not surpassed by any extraction-based pipeline tried here. See "Approaches worth trying next" when resuming.