SAM3-Joint-Surgical

Surgical SAM 3.1

<p align="center"> <a href="https://keremturkcan.com/">Mehmet Kerem Turkcan</a>, Soham Samal, Zoran Kostic <br> <a href="https://www.aidl.ee.columbia.edu/">AIDL Lab, Columbia University</a>, <a href="https://www.i-pal.org/">IPAL Lab</a> </p> <p align="center"> <a href="assets/surgical_sam3_overview.png"><img src="assets/surgical_sam3_overview.png" width="100%" alt="Examples from four surgical datasets with reference masks for instruments and anatomy"></a> </p>

Surgical SAM 3.1 detects and segments surgical instruments, anatomy, and tissue from text prompts. The detector uses one vocabulary across CholecInstanceSeg, CholecSeg8k, the Dresden Surgical Anatomy Dataset, and Endoscapes. Training uses the grouped V3 dataset, which assigns each procedure, video, or patient group to one split.

The released checkpoint contains the fine tuned SAM 3.1 detector and the original SAM 3.1 multiplex tracker parameters. Image detection and instance masks use the fine tuned detector. Video propagation uses the original tracker components after the detector initializes the objects.

<p align="center"> <a href="assets/sam31_results_summary.png"><img src="assets/sam31_results_summary.png" width="100%" alt="Matched grouped test results for Base SAM 3.1 and Surgical SAM 3.1"></a> </p> <p align="center"> <a href="assets/instrument_examples_1.png"><img src="assets/instrument_examples_1.png" width="100%" alt="Reference masks for grasper, hook, irrigator, and clipper"></a> </p> <p align="center"> <a href="assets/instrument_examples_2.png"><img src="assets/instrument_examples_2.png" width="100%" alt="Reference masks for bipolar, scissors, snare, and surgical instrument"></a> </p> <p align="center"> <a href="assets/anatomy_tissue_examples_1.png"><img src="assets/anatomy_tissue_examples_1.png" width="100%" alt="Reference masks for abdominal wall, liver, gastrointestinal tract, fat, connective tissue, and cystic duct"></a> </p> <p align="center"> <a href="assets/anatomy_tissue_examples_2.png"><img src="assets/anatomy_tissue_examples_2.png" width="100%" alt="Reference masks for gallbladder, hepatic vein, liver ligament, cystic plate, Calot triangle, and cystic artery"></a> </p> <p align="center"> <a href="assets/anatomy_tissue_examples_3.png"><img src="assets/anatomy_tissue_examples_3.png" width="100%" alt="Reference masks for colon, pancreas, small intestine, spleen, and stomach"></a> </p>

The galleries visualize reference masks from the grouped dataset. Positive annotations are available for 25 text concepts.

<p align="center"> <a href="assets/sam31_per_concept_results.png"><img src="assets/sam31_per_concept_results.png" width="100%" alt="Box and mask average precision for every category with grouped test instances"></a> </p>

Results

The grouped test split contains 4,134 physical images in 22 groups. COCO AP averages precision across ten IoU thresholds from 0.50 through 0.95 and across the 24 concepts with positive test instances. Each image is evaluated only for the concepts listed in its annotation scope.

Both evaluations use the same prompts, images, annotation scopes, postprocessing, and COCO evaluator.

<details> <summary>Test results by concept</summary> </details>

Model selection

Checkpoint and loss selection use mask AP on 4,144 validation images. The held out test annotations do not determine the selected checkpoint. The selected model starts from epoch 7 and updates the segmentation head with the loss configuration identified by the validation search.

The selected candidate is dice_polish_epoch1, with validation box AP 0.5130 and validation mask AP 0.3387.

Evaluation protocol

Each record contains exhaustive_category_ids, which lists the concepts that are annotated for that image. The evaluator issues only those text prompts and retains predictions for those categories.

The grouped split merges records that resolve to the same physical frame before assignment. Its audit found zero overlap between every pair of splits for group identifiers, physical frame identifiers, normalized frame identifiers, and SHA 256 image digests. Generated Endoscapes tool masks occur only in training.

Quick start

Download the image model builder patch:

hf download mehmetkeremturkcan/SAM3-Joint-Surgical \
  patches/sam31_image_training.patch \
  --revision main \
  --local-dir surgical_sam31

Install the SAM 3 source used for this run:

git clone https://github.com/facebookresearch/sam3.git
cd sam3
git checkout 46957e47805eaa273f4aa7bbbd25a88bca9108ce
git apply ../surgical_sam31/patches/sam31_image_training.patch
pip install -e .

Download the configuration and checkpoint:

hf download mehmetkeremturkcan/SAM3-Joint-Surgical \
  config.json \
  joint_grouped_sam31_best_multiplex.pt \
  --revision main \
  --local-dir checkpoints

Run image detection and segmentation:

import torch
from PIL import Image

from sam3.model.sam3_image_processor import Sam3Processor
from sam3.model_builder import build_sam31_image_model

model = build_sam31_image_model(
    checkpoint_path="checkpoints/joint_grouped_sam31_best_multiplex.pt",
    load_from_HF=False,
    device="cuda",
    eval_mode=True,
    enable_segmentation=True,
)
processor = Sam3Processor(model, device="cuda", confidence_threshold=0.5)
image = Image.open("frame.png").convert("RGB")

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = processor.set_image(image)
    output = processor.set_text_prompt(
        state=state,
        prompt="surgical instrument",
    )

boxes = output["boxes"]
masks = output["masks"]
scores = output["scores"]

The complete example is available in scripts/inference_sam31.py.

Checkpoint

The checkpoint loads with weights_only=True. The unsafe serialization global check reports zero globals. The tracker tensors retain the values from facebook/sam3.1; the detector tensors contain the validation selected surgical model.

Training

The full detector run trains the SAM 3.1 image grounding path while retaining the interactive and propagation neck parameters from the base checkpoint. The refinement search updates only the segmentation head, which isolates mask quality from detector changes. Exact configurations and machine readable metrics are included in this repository.

Dataset

The associated SAM3 Joint Surgical Dataset contains 41,372 physical images with 175,674 annotations across 29 schema concepts. Positive annotations are available for 25 concepts. Train, validation, and test contain 33,094, 4,144, and 4,134 images, respectively.

Intended use

This repository supports research in surgical computer vision. The model has not been validated for clinical decision making or patient care.

Authors

Mehmet Kerem Turkcan, Soham Samal, and Zoran Kostic. This project was developed by the AIDL Lab at Columbia University and IPAL Lab.

Citation

@misc{turkcan2026surgicalsam31,
  title  = {Surgical SAM 3.1},
  author = {Turkcan, Mehmet Kerem and Samal, Soham and Kostic, Zoran},
  year   = {2026},
  url    = {https://huggingface.co/mehmetkeremturkcan/SAM3-Joint-Surgical}
}

Licenses

The checkpoint follows the SAM License in LICENSE. Dataset records retain their source licenses. See the dataset repository for source attribution and redistribution terms.