vjepa2-archetypal-transcoder

V-JEPA 2 Archetypal Sparse Transcoder (block-10 FFN)

To our knowledge, the first sparse transcoder trained on a self-supervised video model's internal features. It sparsely decomposes the computation of the block-10 FFN of a frozen V-JEPA 2.1 ViT-B/16 384px encoder, using an archetypal decoder (RA-SAE-style: decoder atoms are sparse convex combinations of K-means centroids of real activations) combined with the layer-to-layer prediction objective of sparse transcoders.

Headline checkpoint from "V-JEPA 2 Video Transcoders Find Motion-Based Features" (Ghassemlou & Joseph, 1st IJCAI Workshop on Safe Physical AI, 2026).

What it does

The transcoder reads the input to the block-10 FFN (norm2(x), i.e. the post-LayerNorm activation) and predicts that FFN's output, through a sparse latent z:

z      = ReLU(W_enc · x + b_enc)                       # sparse codes  [N, 12288]
D      = topk_softmax(A_logits, K) @ data_pool + Lambda # archetypal decoder atoms
x_hat  = Dᵀ · z + b_dec                                 # predicted FFN output [N, 768]

The K-means pool (data_pool) is bundled inside the weights, so the decoder is fully self-contained.

Held-out performance (2M tokens, clip-disjoint 10% split of SSv2)

A random-init control at the same architecture and same K-means pool (random_init_control.safetensors) scores EV ≈ −1.4×10⁴, L0 ≈ 6,121

Files

Usage

# needs: torch, safetensors, and modeling_archetypal_transcoder.py from this repo
from load_example import load_transcoder
m = load_transcoder("model.safetensors")          # data_pool is inside the weights
x_hat, z = m(x)   # x: V-JEPA-2.1 block-10 FFN *input* activations, [N, 768] fp32

Substrate + preprocessing are part of the model

The weights are only meaningful against the exact activations they were trained on. You must:

1. Run the frozen V-JEPA 2.1 ViT-B/16 384px encoder (Meta's vjepa2_1_vit_base_384; downloaded from Meta's public CDN, its use is governed by Meta's license).
2. Extract the input to blocks[10].mlp (norm2(x)) as the transcoder input; the target it predicts is that FFN's output.
3. Match preprocessing exactly: 384px, 16 frames, tubelet 2, patch 16, fp32, ImageNet normalization (mean=[0.485,0.456,0.406], std=[0.229,0.224,0.225]).

Feeding differently-normalized or wrong-layer activations produces meaningless output. The full extraction / training / evaluation / feature-viewer pipeline is in the code repo below.

Interpretability

Features were analyzed with a non-VLM, label-distribution protocol over the 174 Something-Something v2 motion templates, surfacing motion-aligned features that track a common motion primitive across different object identities. See the paper and code repo for the feature viewer and concentration analysis.

Code

https://github.com/sghassemlou/V-JEPA-2-Transcoder --> training, evaluation, and the interactive feature-viewer (HTML) code.

Citation

@inproceedings{ghassemlou2026vjepa2transcoder,
  title     = {V-JEPA 2 Video Transcoders Find Motion-Based Features},
  author    = {Ghassemlou, Susan Soraya Afshar and Joseph, Sonia},
  booktitle = {1st IJCAI Workshop on Safe Physical AI},
  year      = {2026}
}

License

Transcoder weights released under CC-BY-4.0. Use of the underlying V-JEPA 2 model is governed by Meta's license terms.