eigenvideo-checkpoints

EigenVideo Checkpoints

Checkpoints from EigenVideo - an autoregressive video generation model that predicts future frames in HunyuanVideo's 16-channel latent space.

Architecture Overview

EigenVideo uses the EigenSwitch architecture with:

• EigenTitan blocks for temporal modeling with persistent memory slots
• Mamba2-style SSM for spatial processing
• T5-large text conditioning (1024-dim embeddings)
• Overlapping patch embeddings for improved spatial continuity

The model operates in HunyuanVideo's VAE latent space (16 channels, 8x spatial, 4x temporal compression).

Available Checkpoints

1. Overfit Checkpoint (58.6M params)

File: overfit_step_500.pt

Single-video memorization checkpoint that validates the architecture works. Trained on one video clip to near-perfect reconstruction.

Config:

{
    "d_model": 512,
    "n_layers": 12,
    "n_slots": 4,
    "n_heads": 64,
    "text_embed_dim": 1024,
    "num_cross_attn_heads": 8,
    "use_overlapping_patches": False
}

2. 50M H100-Optimized Checkpoint

File: 50M_h100_step_200.pt

Larger model with H100-optimized settings. Trained on multiple video clips with text conditioning.

Config:

{
    "d_model": 768,
    "n_layers": 16,
    "n_slots": 64,
    "n_heads": 64,
    "headdim": 32,
    "text_embed_dim": 1024,
    "num_cross_attn_heads": 8,
    "use_overlapping_patches": True
}

Usage

import torch
from eigenvideo.models.student.eigen_switch_model import EigenSwitchModel

# Load checkpoint
ckpt = torch.load("overfit_step_500.pt", map_location="cpu")
config = ckpt["config"]

# Create model
model = EigenSwitchModel(**config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# Generate video frames
# z_0: initial latent frame [B, 16, H, W]
# text_embed: T5-large embeddings [B, seq_len, 1024]
output = model(z_0, seq_len=16, text_embed=text_embed)
z_pred = output.predicted_latents  # [B, T, 16, H, W]

Decoding with HunyuanVideo VAE

from diffusers import AutoencoderKLHunyuanVideo

vae = AutoencoderKLHunyuanVideo.from_pretrained(
    "hunyuanvideo-community/HunyuanVideo",
    subfolder="vae",
    torch_dtype=torch.float16,
)
vae = vae.cuda().eval()

# z_pred shape: [B, C, T, H, W] where C=16
z_pred = z_pred / vae.config.scaling_factor
frames = vae.decode(z_pred).sample  # [B, 3, T*4, H*8, W*8]

Training Data

These models were trained on latents from the UltraVideo dataset:

• Videos: 720p, 65 frames @ 24fps
• Latents: HunyuanVideo VAE 16-channel latents
• Text: Captions encoded with T5-large

Related datasets:

• SunsetOnCordoba/ultravideo-720p-65f-sharded - Raw videos
• SunsetOnCordoba/ultravideo-latents-text - Precomputed latents with text embeddings

License

Apache 2.0

Citation

@misc{eigenvideo2024,
  title={EigenVideo: Autoregressive Video Generation with EigenTitan Memory},
  year={2024}
}