Downloads · 30 days
75
9% of all-time downloads
gullalc/videomae-base-finetuned-kinetics-movieshots-scale
videomae-base-finetuned-kinetics-movieshots-scale is a video classification model from gullalc. Use it for the video classification task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
videomae-base-finetuned-kinetics model finetuned to classify shot scale into five classes: ECS (Extreme close-up shot), CS (close-up shot), MS (medium shot), FS (full shot), LS (long shot)
Downloads · 30 days
75
9% of all-time downloads
All-time downloads
806
Public
Parameters
86.2M
690 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors345 MB · 100%
From the Hugging Face model README
videomae-base-finetuned-kinetics model finetuned to classify shot scale into five classes: ECS (Extreme close-up shot), CS (close-up shot), MS (medium shot), FS (full shot), LS (long shot)
Movienet dataset is used for finetuning the model for 5 epochs. v1_split_trailer.json provides the training, validation and test data splits.
Model achieves accuracy of 88.93% and macro-f1 of 89.19%
Class-wise accuracies: ECS - 91.16%, CS - 83.65, MS - 86.2%, FS - 90.74%, LS - 94.55%
This is how model can be tested on a shot/clip from a video. Same code is used to process, transform and evaluate on the movienet test set.
from transformers import VideoMAEImageProcessor, VideoMAEForVideoClassification
from pytorchvideo.transforms import ApplyTransformToKey
from torchvision.transforms import v2
from decord import VideoReader, cpu
## Evaluation Transform
transform = v2.Compose(
[
ApplyTransformToKey(
key="video",
transform=v2.Compose(
[
v2.Lambda(lambda x: x.permute(0, 3, 1, 2)), # T, H, W, C -> T, C, H, W
v2.UniformTemporalSubsample(16),
v2.Resize(resize_to),
v2.Lambda(lambda x: x / 255.0),
v2.Normalize(img_mean, img_std)
]
),
),
]
)
## Preprocessor and Model loading
image_processor = VideoMAEImageProcessor.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-scale")
model = VideoMAEForVideoClassification.from_pretrained("gullalc/videomae-base-finetuned-kinetics-movieshots-scale")
img_mean = image_processor.image_mean
img_std = image_processor.image_std
height = width = image_processor.size["shortest_edge"]
resize_to = (height, width)
## load video/clip and predict
video_path = "random_clip.mp4"
vr = VideoReader(video_path, width=480, height=270, ctx=cpu(0))
frames_tensor = torch.stack([torch.tensor(vr[i].asnumpy()) for i in range(len(vr))]) ## Shape: (T, H, W, C)
frames_tensor = transform({"video": frames_tensor})["video"]
output = model(pixel_values=frames_tensor)
pred = torch.argmax(outputs.logits, axis=1).cpu().numpy()
print(model.config.id2label[pred[0]])