SurgRec

SurgRec — Surgical Video Foundation Models

Pretrained backbones from Scaling Video Pretraining for Surgical Foundation Models.

<p align="center"><em>Each dataset is compared against whichever third-party baseline is strongest on it — DINOv3, DINOv3-SurgeNetXL, VideoMAE, or V-JEPA (colour-coded; VideoMAE is never the strongest). Numbers are Table 2 of the paper. cataract-1k is the one dataset where a baseline leads, and cat-21 is a tie where every method scores 15.69.</em></p>

hf download SichengLu/SurgRec --local-dir checkpoints

These are headless backbones — no classification head. Fine-tune them on a downstream surgical video dataset with the code repository.

Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng, Jian Wu, Zongxin Yang, Zuozhu Liu

Highlights

• Pretrained on a multi-source surgical corpus of 10,535 videos / 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
• Evaluated on a standardized 16-dataset downstream benchmark across four clinical domains with consistent splits.
• SurgRec-MAE is strictly the best of the seven compared methods on 11 of 16 datasets, and ties for best on 3 more. On the mean it leads the strongest third-party baseline (JEPA) by 11.3 points.

Results

Top-1 accuracy (%) on the 16-dataset downstream benchmark, from Table 2 of the paper. Baselines are DINOv3, DINOv3-Surg (SurgeNetXL), V-MAE (VideoMAE), SR-MAE without balanced sampling, and JEPA; bold marks the best method on each dataset.

SR-MAE (w/o bal.) is an ablation of our own model, not a third-party method. Every method scores exactly 15.69 on cat-21, consistent with all of them collapsing to a single class, so that row carries no signal. JEPA leads on cataract-1k and SurgRec-JEPA on Cholec80.

Models

<details> <summary><b>Checkpoint internals, training configuration, and checksums</b></summary>

surgrec_mae.pth — configuration recorded inside the checkpoint:

• model = pretrain_videomae_base_patch16_224
• mask_type = tube, mask_ratio = 0.9, decoder_depth = 4
• input_size = 224, num_frames = 16, sampling_rate = 4
• lr = 3e-4, batch_size = 64, taken at epoch = 149

The file stores 94.2 M parameters in total including the MAE decoder; the encoder used for fine-tuning is 86.5 M. The archive holds model, optimizer, scaler, epoch, and args. The model entry contains both the encoder and the MAE decoder; fine-tuning loads the encoder and discards decoder.*.

surgrec_jepa.pt — holds encoder, predictor, and target_encoder, each ViT-L/16; taken at epoch = 131, loss = 0.4225, trained with batch_size = 24 across world_size = 8. Load the encoder entry; its weights sit under a module.backbone. prefix, which the loader in the code repository strips.

SHA256, for verifying downloads:

ce2c0298292d636f95c77a12c26fcdeb93de8b4a78e515a68d247de5e61759c5  surgrec_mae.pth
84040aa2e2fc015d03165903e6eec06282507f1d275e76efa97292457cd1bd3e  surgrec_jepa.pt

</details>

Usage

git clone https://github.com/Siiichenggg/SurgRec.git
cd SurgRec
pip install -r requirements.txt
hf download SichengLu/SurgRec --local-dir checkpoints

Fine-tune on a downstream dataset:

DATASET_ROOT=data/splits_local CKPT_ROOT=checkpoints GPUS=8 \
  bash scripts/train/finetune_surgrec_video.sh cholec80 --backbone surgrec_mae

Or reproduce a paper run with its original hyperparameters:

CKPT_ROOT=checkpoints DATA_ROOT=/path/to/split GPUS=8 \
  bash scripts/paper/finetune/jepa_v3_finetune_8gpu.sh

License and intended use

Released for non-commercial research. These weights are a research artifact, not a medical device, and must not be used for clinical diagnosis or treatment decisions.

CC BY-NC 4.0 — attribution required, no commercial use. The code derives in part from VideoMAE, which carries the same obligation; see LICENSE and NOTICE in the code repository.

Citation

@article{lu2026surgrec,
  title   = {Scaling Video Pretraining for Surgical Foundation Models},
  author  = {Lu, Sicheng and Xiao, Zikai and Wei, Jianhui and Sun, Danyu and
             Lu, Qi and Hu, Keli and Feng, Yang and Wu, Jian and Yang, Zongxin
             and Liu, Zuozhu},
  journal = {arXiv preprint arXiv:2603.29966},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.29966}
}