Video-4M-tokenizers

Video-4M Tokenizers

🌐 Project page | 📄 Paper | 💻 Code | 🤗 Models | 🤗 Demo

This repository hosts the modality tokenizers used by Video-4M, where each modality is represented as a sequence of discrete tokens using its own dedicated tokenizer.

Tokenizers

Let's briefly go through the tokenization details for each modality:

• RGB / Depth / Surface Normals / Optical Flow: use separate FSQ VAE VidTok tokenizers. They share the same causal VidTok architecture, and each modality tokenizer is finetuned on its modality-specific data.
• SigLIP 2 / DINOv2 / V-JEPA 2: use VidTok tokenizers adapted to accept feature map inputs. We train these feature map tokenizers from scratch.

Video-4M operates on 4-second clips (17 frames sampled at 4 FPS) at 128x128 resolution. For the feature map modalities, features are extracted at each extractor's default resolution before tokenization, i.e., 224x224 for DINOv2 and SigLIP 2, and 256x256 for V-JEPA 2.

The remaining modalities are not hosted here:

• Caption / Transcription / Bounding Boxes share one WordPiece text tokenizer (with special tokens included for quantized bounding-box coordinates and per-frame separators), which is provided directly in our main GitHub code repository.
• 3D Human Poses use the pose VQ-VAE tokenizer from EPFL-VILAB/4M_tokenizers_human-poses_1k_8.
• Class condition uses a learned embedding layer with 600 entries (one per K600 action category), which is pretrained as part of the Video-4M model.

Usage

The notebooks in our main GitHub code repository download these tokenizers automatically. Below, we provide an example of loading the RGB tokenizer directly and using its encode/decode calls:

import torch
from notebooks import pipeline

tokenizers = pipeline.load_tokenizers()  # downloads the tokenizers from the Hub on first call
rgb_tokenizer = tokenizers.modality_config["tok_video_rgb@128"]["tokenizer"]

# A real clip: [0, 1] pixels, resized to 128x128, then normalized to
# roughly [-1, 1] (mean=0.5, std=0.5)
video = torch.randn(1, 3, 17, 128, 128).to(tokenizers.device)  # B C T H W, 17 frames

with torch.no_grad():
    # Encode: video -> discrete RGB tokens
    _, reg_log = rgb_tokenizer.encode(video, return_reg_log=True)
    tokens = reg_log["indices"].reshape(-1)
    print(tokens.shape)  # torch.Size([1280])

    # Decode: tokens -> reconstructed video
    latents = rgb_tokenizer.regularization.indices_to_codes(reg_log["indices"])
    reconstructed = rgb_tokenizer.decoder(latents)
    print(reconstructed.shape)  # torch.Size([1, 3, 17, 128, 128])

If you would like to tokenize your own multimodal video dataset with these tokenizers, please see README_TOKENIZATION.md. Using these tokenizers, we have also publicly released the tokenized K600-MM dataset, which we used to pretrain Video-4M.

Citation

If you find these tokenizers useful, please consider citing:

@article{video4m2026,
    title={{Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities}},
    author={Khattak, Muhammad Uzair and Kim, Won Jun and Abbassi, Reza and Havolli, Albias and Murphy, Michael and Zadeh, Amir and Li, Chuan and Naeem, Muhammad Ferjad and Kar, O\u{g}uzhan Fatih and Bachmann, Roman and Atanov, Andrei and Tombari, Federico and Zamir, Amir},
    journal={arXiv preprint},
    year={2026},
}

Our tokenizers are adapted from VidTok. We thank the authors for releasing their code.