fga-video-retrieval

Video-text matching on YouCook2

Built on the attention layer of Factor Graph Attention (CVPR 2019), following Video and Text Matching with Conditioned Embeddings (WACV 2022). A caption and a video are attended jointly and scored against each other, trained with a max-margin contrastive loss against the hardest negative in the batch.

Code: github.com/idansc/fga.

Read the split before the numbers

This is not YouCook2's official train/val division, and these numbers do not compare to published YouCook2 retrieval figures.

The feature dump available here covers YouCook2's validation videos only -- 436 of them -- so the data was divided over those by video, no video appearing on two sides:

Retrieval over 617 segments is an easier problem than retrieval over the official split's several thousand, and the numbers below should be read as an internal comparison only.

Results

Held-out test slice, 617 segments. Recall at k in both directions:

The checkpoint is the one that scored best on the validation videos, not on the test slice. Selecting on the test slice reads 0.438 / 0.468 instead; the validation split exists so that choice is not made there.

R@10 here counts a hit at rank 10 or better. Worth stating because the reference implementation's evaluation.py computes its r10 as ranks < 50 in all four of its ranking functions -- a commented-out print in the same file labels that value "Ranks@50" -- so an R@10 reported by that code is an R@50 and is not comparable to the column above.

Training

Hidden size 768, learning rate 2e-4, 2,000 epochs. Video features are the InternVideo MM-L14 dump; captions are encoded with GloVe 300d.

Usage

from fga.tasks.video_retrieval import VideoMatchModel

model = VideoMatchModel.from_pretrained("Idan/fga-video-retrieval")
scores = model(caption_input_ids=text, video_features=video).logits

Citation

@inproceedings{ali2022video,
  title={Video and Text Matching with Conditioned Embeddings},
  author={Ali, Ameen and Schwartz, Idan and Hazan, Tamir and Wolf, Lior},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2022}
}

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{zhou2018towards,
  title={Towards Automatic Learning of Procedures from Web Instructional Videos},
  author={Zhou, Luowei and Xu, Chenliang and Corso, Jason J},
  booktitle={AAAI Conference on Artificial Intelligence},
  year={2018}
}