Viggle-Animate

TechnoBaptist/Viggle-Animate

Viggle-Animate is a video-to-video model from TechnoBaptist. Use it for the video-to-video task on the model card, and read the license before you ship it in a product. It is set up for diffusers. The card lists the license as other.

Try the demo  ·  viggle.ai/h3  ·  Built on MiniMaxAI/MiniMax-H3

Downloads · 30 days

Access

Public

Updated Sep 4, 2026

Parameters

33.1B

68.9 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors68.9 GB · 100%

At a glance

Task: Video-to-Video
Library: diffusers
License: other
Access: Public
Created: Sep 4, 2026
Updated: Sep 4, 2026
SHA: 0ab0ddd1

Base models

MiniMaxAI/MiniMax-H3

Model card

From the Hugging Face model README

Viggle-Animate

Character Replacement in Video from a Single Repainted Frame

Try the demo · viggle.ai/h3 · Built on MiniMaxAI/MiniMax-H3

Viggle-Animate replaces the character in a video with whatever you paint into one of its own frames — motion, camera and timing untouched. You prepare that one frame in an image editor; from there the video stage runs no pose estimator, segmenter, face tracker or text encoder. Two inputs, three forward passes, 26 seconds a shot on one GPU.

It is strongest where replacement is hardest: fast motion, and pose transfer accurate enough to follow it. Whipping heads, full kicks, jumps — tracked frame for frame, not smeared through.

Abstract

Controlled character replacement is usually built on intermediate representations — pose skeletons, segmentation masks, background plates, face crops. Each needs its own extractor, and each extractor is another model to run and another place to lose information. Recent work drops the skeleton but keeps a mask channel. Viggle-Animate uses neither. Its two inputs are a driving video and one of that video's own frames with the character repainted, and its only task is to propagate that edit across the shot.

Because the reference is a frame of the clip, its pose, camera, framing and lighting already agree with the footage, and nothing downstream has to align them again. The model is never told what the new character is: no class, no identity encoder, and no user-provided text prompt. Viggle-Animate is a 33.1 B full finetune of MiniMax-H3's ref2va transformer, jointly distilled with DMD to three forward passes. In a matched comparison on the same machine and B200 GPU, using the same source videos, output resolution and frame count, it renders 124 frames in 26 s, 6.1× faster per clip than Wan2.2-Animate-14B.

Method

Character replacement asks two questions at once: what does the new character look like, and how does it move through this shot. Systems that condition on a standalone character photograph must answer both, and reconciling a photograph with footage it was never part of is what the scaffolding exists for.

State-of-the-art image models have finished that job. Give gpt-image a frame and an instruction and it replaces the character while following the prompt exactly — transferring the pose, matching the lighting, preserving the background. The hard reconciliation is already solved, once, on one image. This model is the second half of that pipeline, not the whole of it.

Appearance enters only through the repainted frame; geometry enters only through the driving video. The text encoder is never loaded. Conditioning is one frozen embedding shipped with the weights (assets/fixed_prompt.txt), identical for every render.

Left panel is the driving video, right panel is this model:

It is fast twice over. Once the repainted frame exists there is nothing else to run — no pose estimator, no segmenter, no face tracker, no text encoder. And the sampler is distilled, so a finished clip is three forward passes rather than thirty. The two compound: one model, one GPU, and no orchestration to get wrong.

The distillation is joint, across two teachers split by noise level. Our finetune supervises the high-noise end of the schedule, where the replacement itself is decided — it is the model that gets the swap right. The original MiniMax-H3 supervises the low-noise end, where detail and texture are decided — it is the model with the better image quality. Distilling each end against the teacher that owns it keeps both properties in one student, instead of inheriting the finetune's visual regressions along with its replacement ability.

It generalizes past humans, because nothing in the loop assumes one. A pose skeleton has a neck and two arms; a mask has a person-shaped hole. We have neither, so the model holds no representation that a character must be a person. What it can animate is bounded by what you can paint.

Efficiency

<div align="center"> <div style="display:flex;gap:10px;flex-wrap:wrap;justify-content:center;margin:20px 0 6px;text-align:left"> <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> <div style="font-size:1.7em;font-weight:700;line-height:1.15">26 s</div> <b>per render</b><br> <small style="opacity:.7">124 frames at 24 fps, 480×832, a single B200</small> </div> <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> <div style="font-size:1.7em;font-weight:700;line-height:1.15">3</div> <b>forward passes</b><br> <small style="opacity:.7"><code>--steps 4</code> names four sigma boundaries, so three passes</small> </div> <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> <div style="font-size:1.7em;font-weight:700;line-height:1.15">2</div> <b>inputs</b><br> <small style="opacity:.7">a clip, and one of its own frames repainted</small> </div> <div style="flex:1 1 170px;border:1px solid rgba(128,128,128,.35);border-top:3px solid #00D94F;border-radius:10px;padding:12px 14px"> <div style="font-size:1.7em;font-weight:700;line-height:1.15">0</div> <b>other models</b><br> <small style="opacity:.7">in the video stage — no pose estimator, segmenter, face tracker or text encoder</small> </div> </div> </div>

One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented replacement recipe — 20 steps, sample_shift 5.0, --refert_num 1 --replace_flag --use_relighting_lora, --w_len 1 --h_len 1 — after its own preprocessing pass.

	Viggle-Animate	Wan2.2-Animate-14B
Inputs	driving video + one repainted frame	driving video + character image, then a preprocessing pass producing pose, face, mask and background tracks
Render, after weights load	26 s	160 s
— of which sampling	13.6 s	140 s
Forward passes	3	40 (20 steps × 2 chunks)
Parameters	33.1 B	17.3 B

6.1× faster per render, 10.3× on sampling alone. Wan's preprocessing pass is not counted in its 160 s.

Qualitative comparison

Four panels each: painted reference · driving video · this model · Wan2.2-Animate-14B, the last at its documented replacement settings. The gap is widest under fast motion: where the comparison smears, this model stays sharp and lands the pose on the right frame.

Generalization

The model is never told what it is animating, so how far the character can get from a person is an empirical question rather than a list of supported categories. Three panels each: painted reference · driving video · this model. Every clip below is one paint and one render at the shipped defaults, --seed 42 — no best-of-N.

Animals. Ears, eye patches and flippers move on limbs the driving clip does not have — the paint places them, the render animates them as if they had always been arms and a head.

Not humanoid. The airliner is the hardest case we have: the paint binds wings to arms and landing gear to legs, and the model's job is to keep that binding for 124 frames. The robot has to relight specular metal as it turns.

Stylized. The clay figure holds its style boundary for the whole clip.

More than one character. The work moves into the paint prompt, which has to bind each one to a position — "the one on the left". The last clip is a wide arena shot, each figure a few dozen pixels tall.

Limitations

It inherits the image edit. What the paint does not show, the model will not add, and where paint and video disagree the video wins. Appearance comes from the paint but shape comes from the driving pose: a LEGO minifigure kept its palette and yellow claw hands, yet reverted to human anatomy — the airliner held because the paint tied its wings to real arms.

Lip-sync is weak. Mouth shapes do not track speech closely in close-ups. Identity and expression hold; it is the sync that lags, and we believe that is a training-data limit rather than anything structural.

Complex scenes are harder than single subjects. Several characters at once, close interaction between them, and shots that cut are all cases where quality drops off — enough that we would not call them solved.

We are training a substantially better model right now, aimed squarely at these three. This release is the version we can ship today, not the ceiling.

What this repository contains

Two parts, both derived from MiniMaxAI/MiniMax-H3's ref2va transformer:


`transformer/`	33.1 B, bf16, 14 shards. A full finetune of the base `transformer_ref` on a character-replacement objective
`lora/`	rank 128 over 302 linear layers, 2.5 GB. A DMD2-distilled delta on that finetune — this is what collapses the sampler to three forward passes

The LoRA is a delta on the finetuned transformer — loading it onto stock transformer_ref produces garbage.

Quickstart

This repository ships only the transformer and the LoRA — the VAE, audio VAE and schedulers load from your own copy of the base model. Inference touches 11 GB of its 269 GB:

hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
  --include "modular_model_index.json" "vae/*" "audio_vae/*" \
            "scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4"

hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate

pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av

python Viggle-Animate/inference/sample.py \
  --model-dir ./MiniMax-H3 \
  --cond driving.mp4 --ref ref.png --out swapped.mp4

d6726f3 is the tested diffusers commit; the upstream minimax_h3 modular pipeline is enough, no fork or patch. The last --include is the clip examples/demo.sh needs.

One 80 GB card is not enough at bf16 — the transformer is 62 GiB resident and a 480×832 / 124-frame render peaks at 80.1 GiB allocated. Use a card with ≥ 96 GB, or pass --offload to stream blocks from CPU (~12 GB resident, much slower).

It also runs quantized on consumer hardware. We deploy it on a single RTX 5090 (32 GB): NVFP4 weights — 4.5 bits/param, dispatching to the real sm_120 cutlass block-scaled kernel, 2.70× bf16 per compiled linear — plus a low-rank adaln_proj and torch.compile. Quantization alone is not enough for 32 GB: 13.0 B of the 33.1 B parameters sit in adaln_proj, which the linear-layer quantizer does not touch. That deployment path is not shipped in this repository.

Defaults are the evaluated configuration: --steps 4 --flow-shift 3 --num-frames 124 (≈ 5.2 s at 24 fps) --seed 42. Output geometry follows the driving clip and must be a multiple of 32 on both axes. Weights load in ~21 s, once per process. Four steps is the operating point, not a shortcut — the distilled model already renders sharper than its teacher, and raising the step count tips that into over-sharpening. --steps 4 is four sigma boundaries and therefore three forward passes. The driving clip's audio is dropped at input; the model emits its own track, and the fixed prompt asks for silence.

Pull a frame with ffmpeg -ss 1.5 -i driving.mp4 -frames:v 1 ref.png and edit it at the same resolution. It does not have to be the first frame. The still is passed to the model with no frame index, so nothing downstream knows where in the clip it came from — pick whichever frame shows the character most clearly, front-on and unoccluded. Name the change, and pin down what must not change: pose, hands, props, framing, background, light. An editor that quietly reframes the shot will fight the driving motion. Prefer clips that keep one side to camera, and bind any new limb to a real one.

examples/demo.sh runs a swap end to end on a clip that ships with the base model, so it needs no media from you — and it is the fastest way to check that the LoRA loaded.

Citation

@misc{viggle2026animate,
  title  = {Viggle-Animate: Character Replacement in Video from a Single Repainted Frame},
  author = {Viggle Research},
  year   = {2026},
  url    = {https://huggingface.co/Viggle/Viggle-Animate}
}

License

The weights are a Model Derivative of MiniMax H3, so the MiniMax H3 Community License applies to them — read it before you redistribute them or ship a product on them. Our changes are listed in MODIFICATIONS.md.

The code in inference/ and examples/ is Apache 2.0 (LICENSE-CODE).

Music in the teaser at the top of this page: "Electrodoodle" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0.

Intended use

This model exists to put a consenting performer into footage they did not shoot, and it will just as readily put someone into footage they never agreed to appear in. Note where that decision is made: the identity comes from the frame you paint, so an image editor's safeguards are upstream of this model and none of them are in it. It cannot verify identity or consent. Do not run it on people who have not agreed to it, label what you generate as AI-generated, and see Section V.5 of the Agreement if you offer this as a service.

Task: Video-to-Video
Library: diffusers
License: other
Created: Sep 4, 2026
Updated: Sep 4, 2026