Motif-VAE

Motif VAE

Motif VAE is a 3D causal video VAE with 4x temporal and 32x spatial compression and a 128-channel latent space, built as a high-compression tokenizer for latent video diffusion models. A 256x256x17-frame clip encodes to a (128, 5, 8, 8) latent, which is 1/16 the spatio-temporal tokens of an 8x-spatial VAE, so downstream diffusion-transformer training and inference cost less. The decoder is much larger than the encoder (about 4:1 in parameters), which helps it keep fine detail such as small text in documents.

It also runs as a plain image VAE at T=1. In its compression class it has the best document text fidelity, video reconstruction, and diffusability (how well a diffusion model converges on its latents) of the tokenizers we compared.

Architecture

<p align="center"> <img src="assets/vae_architecture.png" width="640" alt="MotifVAE architecture: a 5-stage convolutional encoder with a parameter-free Global Skip (pixel unshuffle + channel average) at every stage compresses the clip into a deterministic latent; a 4.14x-larger 5-stage decoder mirrors it back to pixels."> </p>

Usage

The model code ships inside this repository (modeling_motifvae.py) and loads through the stock diffusers AutoModel — no extra package beyond torch, diffusers, einops:

import torch
from diffusers import AutoModel

vae = AutoModel.from_pretrained("Motif-Technologies/Motif-VAE", trust_remote_code=True)
vae = vae.to("cuda", dtype=torch.bfloat16).eval()

# video: (B, 3, T, H, W) in [-1, 1], T = 4n+1 (1, 5, 9, 13, 17, ...),
# H/W multiples of 32.  T=1 works as a plain image VAE.
x = torch.randn(1, 3, 17, 256, 256, device="cuda", dtype=torch.bfloat16)
with torch.no_grad():
    z = vae.encode(x).latent_dist.sample()   # (1, 128, 5, 8, 8)
    rec = vae.decode(z).sample               # (1, 3, 17, 256, 256)

For diffusion training, whiten the latent to roughly unit variance with the per-channel statistics shipped in the config — shift is the channel mean and scale the channel std (128 values each, measured over Kinetics-400 video (256x256x17) and ImageNet-1K images (256x256)). Apply per channel, reshaping both to (1, 128, 1, 1, 1):

z = vae.encode(x).latent_dist.sample()
shift = torch.tensor(vae.config.shift, device=z.device).view(1, -1, 1, 1, 1)
scale = torch.tensor(vae.config.scale, device=z.device).view(1, -1, 1, 1, 1)
z = (z - shift) / scale          # whiten for the diffusion model
# ... sample/denoise in this space, then invert before decoding:
z = z * scale + shift
rec = vae.decode(z).sample

Recompute these statistics on your own data if its distribution differs from natural video.

Evaluation

Baseline numbers are either the published values cited in each table or measured by us under the same protocol. MotifVAE is a 32x spatial tokenizer, so the comparison is against the other 32x tokenizers (LTX-Video, LTX-2, and the f32 image VAEs below); the lower-compression 16x Wan2.2 appears only as a reference upper bound.

Image reconstruction — ImageNet-1K val (256x256) & FFHQ (1024x1024)

All models below are 32x (f32) image VAEs. MotifVAE is measured by us with per-image PSNR / SSIM (the published-table convention); baseline rows are the published numbers from the Qwen-Image-VAE-2.0 technical report (Table 2, same ImageNet-256 / FFHQ protocol). Our own runs of the baselines reproduce those values to within 0.2 dB, so the comparison is direct.

At the f32c128 channel budget MotifVAE leads both benchmarks among 32x image VAEs, above Qwen-Image-VAE-2.0's f32c128 (29.69 / 35.91 dB), LTX-Video, LTX-2, and the lighter f32c64 / f32c32 models.

Qualitative comparison (real photographs)

GT vs frozen-tokenizer reconstruction on ImageNet-1K val photographs; the red box marks the zoom strip shown below each row.

These rows are high-frequency patterns that high-compression VAEs find hard: an armadillo's banded shell, a dense solar-panel grid, brickwork, and vehicle markings. LTX-Video and LTX-2 (same 32x class) lose the pattern, and the solar grid collapses for them entirely. MotifVAE keeps it and stays close to the 16x Wan2.2 reference, which also struggles on some of these.

Document text fidelity — OmniDoc-TokenBench (256x256)

f32-compression VAEs on OmniDoc-TokenBench (~3K text-rich documents). Baseline rows are the published numbers from the Qwen-Image-VAE-2.0 technical report (Table 3, same benchmark / resolution / OCR-NED protocol); sorted by NED.

MotifVAE has the highest OCR-NED of any f32 VAE here: +0.12 over Qwen's f32c128 (same channel budget) and about 2.3x that of LTX-Video / LTX-2. It also leads every pixel metric in its class. The figure below covers English print, Korean print, and Korean handwriting; each row shows the full page and a zoom-in on the red-boxed region. The top two rows are OmniDoc-TokenBench (English); the bottom two are Korean printed text and handwriting from AI-Hub — Korea's national AI-data hub (operated by NIA, the National Information Society Agency), which publishes large-scale Korean AI training datasets (the table metrics above are OmniDoc-only):

Video reconstruction — Kinetics-400 (256x256) & OpenVid-1M (720p)

The same four frozen tokenizers on both benchmarks, under one pipeline: the three 32x spatial tokenizers (MotifVAE, LTX-Video, LTX-2) and the 16x Wan2.2 as a reference.

Kinetics-400 val — 17x256x256

OpenVid val-1000 — native 720p, full-frame

Among the 32x spatial tokenizers MotifVAE leads both tables: +2.7 dB over LTX-Video and +7.3 dB over LTX-2 on K400, and +2.6 / +4.0 dB on 720p. The 16x Wan2.2 reference is a little ahead; the gap to the other 32x VAEs is largest on high-motion clips (examples below).

Protocol. K400: 256 random val clips. OpenVid: the pinned public Dev-Jahn/OpenVid-1M-wds val split (1000 clips), each resized to height 720 keeping aspect (full-frame). Both use 17 frames/clip and report per-frame-averaged PSNR / SSIM / LPIPS under one pipeline; OpenVid LPIPS uses AlexNet. Absolute values aren't comparable across papers.

Video reconstruction examples

Frozen-tokenizer reconstruction on high-motion Kinetics-400 clips. Each clip is one strip: Input / MotifVAE / LTX-Video / LTX-2 / Wan2.2 (labelled). MotifVAE keeps the fast motion close to the 16x Wan2.2; LTX-Video and LTX-2 blur it.

<video controls autoplay loop muted src="https://huggingface.co/Motif-Technologies/Motif-VAE/resolve/main/assets/hero_video_recon.mp4"></video>

<video controls autoplay loop muted src="https://huggingface.co/Motif-Technologies/Motif-VAE/resolve/main/assets/video_recon_1.mp4"></video>

<video controls autoplay loop muted src="https://huggingface.co/Motif-Technologies/Motif-VAE/resolve/main/assets/video_recon_2.mp4"></video>

Diffusability — UCF101 class-conditional generation

Latte-XL/1 (patch-1, rectified flow, 250K steps, no-CFG FVD-2048) over each frozen tokenizer, under the Latte-official stride-3 protocol. The Wan2.2 control (FVD 204) is in line with values reported for this setting in the literature, confirming the harness reproduces the expected scale.

Among the 32x spatial tokenizers MotifVAE leads at every milestone, finishing 39% below LTX-2 and 53% below LTX-Video. It reaches LTX-2's final 250K FVD by about 75K steps (~3x faster) and stays close to the 16x Wan2.2 reference. Reconstruction quality and diffusability are separate properties: LTX-Video reconstructs better than LTX-2 but is the worst of the three for diffusion.

License

MIT