sam2.1_hiera_tiny

Model Card for SAM 2: Segment Anything in Images and Videos

Repository for SAM 2: Segment Anything in Images and Videos, a foundation model towards solving promptable visual segmentation in images and videos from FAIR. See the SAM 2 paper for more information.

Model Details

Model Description

SAM 2 (Segment Anything Model 2) is a foundation model developed by Meta FAIR for promptable visual segmentation across both images and videos. It extends the capabilities of the original SAM by introducing a memory-driven, streaming architecture that enables real-time, interactive segmentation and tracking of objects even as they change or temporarily disappear across video frames. SAM 2 achieves state-of-the-art segmentation accuracy with significantly improved speed and data efficiency, outperforming existing models for both images and videos.

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Meta FAIR (Meta AI Research), Authors: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer.
• Shared by [optional]: Sangbum Choi and Yoni Gozlan
• Model type: Transformer-based promptable visual segmentation model with streaming memory module for videos.
• License: Apache-2.0, BSD 3-Clause

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/facebookresearch/sam2
• Paper [optional]: https://arxiv.org/abs/2408.00714
• Demo [optional]: https://ai.meta.com/sam2/

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Direct Use

SAM 2 is designed for:

Promptable segmentation—select any object in video or image using points, boxes, or masks as prompts.

Zero-shot segmentation—performs strongly even on objects, image domains, or videos not seen during training.

Real-time, interactive applications—track or segment objects across frames, allowing corrections/refinements with new prompts as needed.

Research and industrial applications—facilitates precise object segmentation in video editing, robotics, AR, medical imaging, and more.

Bias, Risks, and Limitations

Generalization Limits: While designed for zero-shot generalization, rare or unseen visual domains may challenge model reliability.

Recommendations

Human-in-the-loop review is advised for critical use cases.

Users should evaluate and possibly retrain or fine-tune SAM 2 for highly specific domains.

Ethical and privacy considerations must be taken into account, especially in surveillance or sensitive settings.

How to Get Started with the Model

from transformers import (
    Sam2Config,
    Sam2ImageProcessorFast,
    Sam2MaskDecoderConfig,
    Sam2MemoryAttentionConfig,
    Sam2MemoryEncoderConfig,
    Sam2Model,
    Sam2Processor,
    Sam2PromptEncoderConfig,
    Sam2VideoProcessor,
    Sam2VisionConfig,
)

image_processor = Sam2ImageProcessorFast()
video_processor = Sam2VideoProcessor()
processor = Sam2Processor(image_processor=image_processor, video_processor=video_processor)

sam2model = Sam2Model.from_pretrained("danelcsb/sam2.1_hiera_tiny").to("cuda")

# `video_dir` a directory of JPEG frames with filenames like `<frame_index>.jpg`
# Try to load your custom video in here
video_dir = "./videos/bedroom"

# scan all the JPEG frame names in this directory
frame_names = [
    p for p in os.listdir(video_dir)
    if os.path.splitext(p)[-1] in [".jpg", ".jpeg", ".JPG", ".JPEG"]
]
frame_names.sort(key=lambda p: int(os.path.splitext(p)[0]))

videos = []
for frame_name in frame_names:
    videos.append(Image.open(os.path.join(video_dir, frame_name)))
inference_state = processor.init_video_session(video=videos, inference_device="cuda")
inference_state.reset_inference_session()

ann_frame_idx = 0  # the frame index we interact with
ann_obj_id = 1  # give a unique id to each object we interact with (it can be any integers)
points = np.array([[210, 350]], dtype=np.float32)
# for labels, `1` means positive click and `0` means negative click
labels = np.array([1], np.int32)

# Let's add a positive click at (x, y) = (210, 350) to get started
inference_state = processor.process_new_points_or_box_for_video_frame(
    inference_state=inference_state,
    frame_idx=ann_frame_idx,
    obj_ids=ann_obj_id,
    input_points=points,
    input_labels=labels
)
any_res_masks, video_res_masks = sam2model.infer_on_video_frame_with_new_inputs(
    inference_state=inference_state,
    frame_idx=ann_frame_idx,
    obj_ids=ann_obj_id,
    consolidate_at_video_res=False,
)

Training Details

Training Data

Trained using a data engine that collected the largest known video segmentation dataset, SA-V (Segment Anything Video dataset), via interactive human-model collaboration.

Focused on full objects and parts, not restricted by semantic classes.

Training Procedure

Preprocessing: Images and videos processed into masklets (spatio-temporal masks); prompts collected via human and model interaction loops.

Training regime: Used standard transformer training routines with enhancements for real-time processing; likely mixed precision for scaling to large datasets.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Evaluated on SA-V and other standard video and image segmentation benchmarks.

Metrics

Segmentation accuracy (IoU, Dice). Speed/Throughput (frames per second).

SAM 2.1 checkpoints

The table below shows the improved SAM 2.1 checkpoints released on September 29, 2024.

SAM 2 checkpoints

The previous SAM 2 checkpoints released on July 29, 2024 can be found as follows:

Results

Video segmentation: Higher accuracy with 3x fewer user prompts versus prior approaches.

Image segmentation: 6x faster and more accurate than original SAM.

Citation [optional]

BibTeX:

@article{ravi2024sam2, title={SAM 2: Segment Anything in Images and Videos}, author={Nikhila Ravi and Valentin Gabeur and Yuan-Ting Hu and Ronghang Hu and Chaitanya Ryali and Tengyu Ma and Haitham Khedr and Roman R{"a}dle and Chloe Rolland and Laura Gustafson and Eric Mintun and Junting Pan and Kalyan Vasudev Alwala and Nicolas Carion and Chao-Yuan Wu and Ross Girshick and Piotr Doll'ar and Christoph Feichtenhofer}, journal={arXiv preprint arXiv:2408.00714}, year={2024} }

APA:

Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., & Feichtenhofer, C. (2024). SAM 2: Segment Anything in Images and Videos. arXiv preprint arXiv:2408.00714.

Model Card Authors [optional]

Sangbum Choi and Yoni Gozlan

Model Card Contact

Meta FAIR (contact via support@segment-anything.com)