vmr3D

vmr3D — Volumetric Motion Rigging

Author and contact: redazul

Experimental assistance: GPT-6-Astra assisted during selected experimental runs. Research direction, final result selection, and Unity validation were provided by redazul.

VMR is a practical workflow for recovering character motion from reconstructed mesh sequences, rebuilding difficult surface detail, and preserving the quality of the final animation.

Release status: research artifacts and preparation for a learned model. This repository currently contains an animated character, demonstrations, validation records, and a proposed learning roadmap. It does not yet contain trained VMR neural-network weights, an implemented learning architecture, or a general video-to-character inference endpoint. The FBX is an animation asset, not a neural-network checkpoint.

What VMR does

Reconstructing a video into independent 3D meshes can reproduce the movement while allowing the character's proportions, grip and surface detail to change between frames. VMR addresses that problem by separating a coarse motion representation from detailed surface reconstruction, then protecting accepted geometry during refinement.

The demonstrated input is a video together with reconstructed reference meshes. The result is a textured character animation that can be opened and inspected in Blender and imported into Unity as an animated FBX. The reference meshes are reconstructed estimates, rather than perfect ground truth.

How the workflow works

1. Establish the reference character. Align the reconstructed targets in a common coordinate system and select detailed reference geometry. Treat camera/alignment differences separately from actual character motion.
2. Recover motion with interior volumes. Partition the reference interior into consistent regions and use interior-distance influence weights to guide the surface. In this implementation, 64 volume controls carry rotation, translation and regularized scale, with four influences per original vertex. These controls are not the same as the skeletal approximation tested later.
3. Fit the surface while preserving local shape. Correct geometric mismatch while protecting accepted regions. A lower nearest-surface error can still damage a palm, collapse a wrist or pull one contacting surface toward another, so distance alone is insufficient.
4. Reconstruct difficult detail explicitly. Recover the hand/grip/bat detail from reference geometry where deformation cannot preserve it. Transport the opening grip coherently through the motion and retain the approved closing configuration.
5. Preserve volume and endpoints. Repair local hand/forearm geometry with thickness constraints. Hold the opening and closing checkpoints fixed and validate them with consistent camera views.
6. Export and check playback. Preserve the reviewed poses in Blender and FBX, compare evaluated geometry after export, and inspect the resulting animation in the target application.

The contribution is an implemented combination of established techniques with explicit reconstruction and validation steps. VMR is a descriptive name for that workflow; scientific novelty or general superiority is not claimed.

What the comparisons established

Two Trimesh non-rigid registration refinements—Amberg and Sumner/Popovic—were evaluated from the existing animation. Both reduced the measured one-way distance to sampled target geometry. The original VMR result was still preferred visually for its lower apparent jitter. The distance measurement uses samples involved in fitting; it is neither a held-out accuracy test nor a temporal jitter score.

A separate Dem Bones comparison approximated a corresponding 51-pose surface using 64 bones and up to four influences per vertex. It demonstrated an actual skeletal approximation, but it did not resolve the separate closing surface. The downloadable VMR character retains the approved baked motion rather than substituting this approximation.

The evidence is specific to this sequence and the tested configurations. Preserving coherent motion can be preferable to following small inconsistencies in each reconstructed target, but the individual causes of the visual differences have not been isolated.

See the result

<video controls playsinline muted src="https://huggingface.co/DiscoSea/vmr3D/resolve/main/whitepaper/assets/vmr_source_reference_result.mp4" width="100%"></video>

Watch the synchronized source / reference mesh / VMR comparison

Input video · Final VMR animation · Actual Blender workspace animation

The Blender demonstration uses real interface captures at each of the 52 stored poses, replayed at 30 fps. It shows editable scene content and timeline poses, not a real-time performance benchmark.

Downloads and the full paper

• Download animated FBX
• Download Blender scene
• Read the complete white paper here on Hugging Face
• Read the interactive white paper on DiscoSea
• Read the PDF on Hugging Face
• Download standalone HTML
• FBX validation record

Both locations contain the full paper. The website adds synchronized video controls; the Hugging Face edition includes figures and direct links to repository-hosted videos.

Use the release

Download the FBX directly for your 3D application. To inspect the complete release locally:

pip install huggingface_hub
hf download DiscoSea/vmr3D --local-dir vmr3D
cd vmr3D
python scripts/verify_release.py
python -m http.server 8080 --directory whitepaper

Open http://localhost:8080/ for the standalone paper. Its media links use Hugging Face and require an internet connection; the corresponding media files are also present in whitepaper/assets/. This serves an existing demonstration; it does not run new reconstruction. The current release provides no automatic Hugging Face inference widget.

For an Animator Controller in Unity, import the FBX as Generic animation, enable blend shapes and animation, and disable animation compression. Add the imported clip to a controller on the model root. The final-surface handoff can be interpolated between sampled poses; no Unity helper package is required or included. Blender playback is available through the supplied scene.

Preparing the learned model

The intended next capability is video → consistent character representation → motion and local corrections → editable animation. The first proposed training task is deliberately bounded: predict temporally consistent VMR controls and surface corrections from a reference character and a video segment. The existing fitting workflow can act as a teacher and refine early predictions.

The intermediate artifacts have several possible uses:

Many meshes from one sequence do not provide the diversity of many independent characters and motions. Rejected outputs need failure labels; they should not all be treated as correct targets. The final-surface handoff must be represented explicitly until continuous correspondence is solved.

A training program would combine synthetic videos from known animated assets with reviewed reconstructions of varied videos. Entire identities and source motions should be held out for evaluation, keeping neighboring frames and all variants of the same source in one split. Dense predicted motion could later be reduced to editable keyframes within a documented error tolerance.

See the learning roadmap and proposed data contract. These are plans; no VMR neural training run, trained weights or generalization results are reported in this release.

Reproducibility and scope

release_manifest.json records SHA-256 hashes and sizes for the release files. The verification script checks file integrity, FBX format signature, linked local paper assets, and the example record. It does not rerun geometric fitting or certify a new model's quality. The original reconstruction scripts and their complete dependency environment are not packaged in this initial release.

Training data: no neural training run is reported in this release. Evaluation: one 52-pose stylized baseball swing, geometric checks, rendered comparisons, qualitative motion review and author-confirmed Unity use. Failure modes include ambiguous contact, stiff rigidly transported detail, imperfect wrist attachments, and the separate final surface.

Related work and license

The workflow builds on established deformation, registration and skinning ideas. Related implementations include TRELLIS.2, Trimesh and Dem Bones. The paper cites the underlying methods. AniGen is a candidate for future comparison, not an evaluated dependency of this release.

The repository retains its author's Apache-2.0 license designation. Third-party methods, models and source assets retain their applicable terms; this repository does not relicense them. See PROVENANCE.md.