Downloads · 30 days
28
3% of all-time downloads
gullalc/videomae-base-finetuned-kinetics-movieshots-movement
videomae-base-finetuned-kinetics-movieshots-movement is a video classification model from gullalc. Use it for the video classification task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
videomae-base-finetuned-kinetics model finetuned to classify shot movement into five classes: Static, Motion, Pull, Push
Downloads · 30 days
28
3% of all-time downloads
All-time downloads
1.1K
Public
Parameters
86.2M
345 MB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors345 MB · 100%
From the Hugging Face model README
videomae-base-finetuned-kinetics model finetuned to classify shot movement into five classes: Static, Motion, Pull, Push
Movienet dataset is used for finetuning the model for 5 epochs. v1_split_trailer.json provides the training, validation and test data splits.
Model achieves accuracy of 91.35% and macro-f1 of 79.37%
Class-wise accuracies: Static - 92.94%, Motion - 91.7%, Pull - 51.25%, Push - 62.73%
This is how model can be tested on a shot/clip from a video. Same code is used to process, transform and evaluate on the movienet test set.
from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
from pytorchvideo.transforms import ApplyTransformToKey
from torchvision.transforms import v2
from decord import VideoReader, cpu
## Evaluation Transform
transform = v2.Compose(
[
ApplyTransformToKey(
key="video",
transform=v2.Compose(
[
v2.Lambda(lambda x: x.permute(0, 3, 1, 2)), # T, H, W, C -> T, C, H, W
v2.UniformTemporalSubsample(16),
v2.Resize(resize_to),
v2.Lambda(lambda x: x / 255.0),
v2.Normalize(img_mean, img_std)
]
),
),
]
)
## Preprocessor and Model loading
image_processor = VideoMAEImageProcessor.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-movement")
model = VideoMAEForVideoClassification.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-movement")
img_mean = image_processor.image_mean
img_std = image_processor.image_std
height = width = image_processor.size["shortest_edge"]
resize_to = (height, width)
## load video/clip and predict
video_path = "random_clip.mp4"
vr = VideoReader(video_path, width=480, height=270, ctx=cpu(0))
frames_tensor = torch.stack([torch.tensor(vr[i].asnumpy()) for i in range(len(vr))]) ## Shape: (T, H, W, C)
frames_tensor = transform({"video": frames_tensor})["video"]
output = model(pixel_values=frames_tensor)
pred = torch.argmax(outputs.logits, axis=1).cpu().numpy()
print(model.config.id2label[pred[0]])