TC-CLIP

<h3 align="center"><a href="https://arxiv.org/abs/2404.09490">[ECCV 2024] Leveraging Temporal Contextualization for Video Action Recognition</a></h3> <div align="center"> <img width="1000" alt="teaser" src="https://cdn-uploads.huggingface.co/production/uploads/66e345c9596fcff3e4b22e5a/O4kbJ5psnH78cTYMPQ3xT.png"> </div> <h5 align="center"> Official model checkpoints for TC-CLIP </h5> <h5 align="center"> If you like our project, please give us a star ⭐ on <a href="https://github.com/naver-ai/tc-clip">Github</a> for the latest update. </h5>

Introduction

We present Temporally Contextualized CLIP (TC-CLIP): A novel video understanding framework that leverages holistic video information within its encoding process.

1. Temporal Contextualization (TC): Unlike prior approaches that access only a limited amount of tokens, TC allows global interactions by summarizing informative tokens from the entire video into context tokens and leveraging them during the feature encoding process.
2. Video-conditional Prompting (VP): Based on the summarized context tokens from the visual domain, VP generates instance-level textual prompts that compensate for the lack of textual semantics in action recognition datasets.
3. Solid performance: TC-CLIP achieves stat-of-the-art performance across zero-shot, few-shot, base-to-novel, fully-supervised settings on five video action recognition benchmarks.

This repository contains all model checkpoints used in our experiments.

Models

We use CLIP ViT-B/16 for all experiments below.

• (LLM) denotes that the models are using LLM-rephrased category names from FROSTER. Note that experiments on the SSv2 dataset do not involve LLM-rephrasing.
• (P) denotes that the models are first pretrained on Kinetics-400 and subsequently fine-tuned on each dataset. Otherwise, models are directly fine-tuned from CLIP. See Appendix A in the paper.

Zero-shot action recognition

Few-shot action recognition

Base-to-novel generalization

Fully-supervised action recognition

Citation

If you find TC-CLIP useful in your research, please consider citing our paper:

@article{kim2024tcclip,
  title={Leveraging Temporal Contextualization for Video Action Recognition},
  author={Kim, Minji and Han, Dongyoon and Kim, Taekyung and Han, Bohyung},
  journal={European Conference on Computer Vision (ECCV)},
  year={2024}
}