SU-T

When Trackers Date Fish: A Benchmark and Framework for Underwater Multiple Fish Tracking

The official implementation of the paper：

When Trackers Date Fish: A Benchmark and Framework for Underwater Multiple Fish Tracking
Weiran Li, Yeqiang Liu, Qiannan Guo, Yijie Wei, Hwa Liang Leo, Zhenbo Li* [Project] [Paper] [Code]

<div align="center"> <img src="assets/Fig.PNG" width="900"/> </div>

Contact: [email protected] or [email protected]. Any questions or discussion are welcome!

If like this work, a star 🌟 would be much appreciated!

---

📌Updates

• [2025.07] Paper released to arXiv.
• [2025.07] Fixed bugs.
• [2025.04] We have released the MFT25 dataset and codes of SU-T!

---

💡Abstract

Multiple object tracking (MOT) technology has made significant progress in terrestrial applications, but underwater tracking scenarios remain underexplored despite their importance to marine ecology and aquaculture. We present Multiple Fish Tracking Dataset 2025 (MFT25), the first comprehensive dataset specifically designed for underwater multiple fish tracking, featuring 15 diverse video sequences with 408,578 meticulously annotated bounding boxes across 48,066 frames. Our dataset captures various underwater environments, fish species, and challenging conditions including occlusions, similar appearances, and erratic motion patterns. Additionally, we introduce Scale-aware and Unscented Tracker (SU-T), a specialized tracking framework featuring an Unscented Kalman Filter (UKF) optimized for non-linear fish swimming patterns and a novel Fish-Intersection-over-Union (FishIoU) matching that accounts for the unique morphological characteristics of aquatic species. Extensive experiments demonstrate that our SU-T baseline achieves state-of-the-art performance on MFT25, with 34.1 HOTA and 44.6 IDF1, while revealing fundamental differences between fish tracking and terrestrial object tracking scenarios. MFT25 establishes a robust foundation for advancing research in underwater tracking systems with important applications in marine biology, aquaculture monitoring, and ecological conservation.

🏆Contributions

• We introduce MFT25, the first comprehensive multiple fish tracking dataset featuring 15 diverse video sequences with 408,578 meticulously annotated bounding boxes across 48,066 frames, capturing various underwater environments, fish species, and challenging conditions including occlusions, rapid direction changes, and visually similar appearances.

• We propose SU-T, a specialized tracking framework featuring an Unscented Kalman Filter (UKF) optimized for non-linear fish swimming patterns and a novel Fish-Intersection-over-Union (FishIoU) matching that accounts for the unique morphological characteristics and erratic movement behaviors of aquatic species.

• We conduct extensive comparative experiments demonstrating that our tracker achieves state-of-the-art performance on MFT25, with 34.1 HOTA and 44.6 IDF1. Through quantitative analysis, we highlight the fundamental differences between fish tracking and land-based object tracking scenarios.

🛠️Installation Guide

Prerequisites

• CUDA >= 10.2
• Python >= 3.7
• PyTorch >= 1.7.0
• Ubuntu 18.04 or later (Windows is also supported but may require additional setup)

Step-by-Step Installation

1. Clone the Repository

   git clone https://github.com/vranlee/SU-T.git
   cd SU-T
   

2. Create and Activate Conda Environment

   # Create environment from yaml file
   conda env create -f conda_env.yaml
   
   # Activate the environment
   conda activate su_t
   

3. Download Required Resources

   • Download pretrained models from BaiduYun (Password: 9uqc)
   • Download MFT25 dataset from BaiduYun (Password: wrbg)

4. Organize the Directory Structure

   SU-T/
   ├── pretrained/
   │   └── Checkpoint.pth.tar
   ├── MFT25/
   │   ├── train/
   │   └── test/
   └── ...
   

🍭Usage Guide

Training

1. Basic Training Command

   python tools/train.py \
       -f exps/SU-T.py \        # Base model configuration
       -d 8 \                   # Number of GPUs
       -b 48 \                  # Batch size
       --fp16 \                 # Enable mixed precision training
       -o \                     # Enable occupy GPU memory
       -c pretrained/Checkpoint.pth.tar  # Path to pretrained weights
   

2. Training with ReID Module

   python tools/train.py \
       -f exps/SU-T-ReID.py \   # ReID model configuration
       -d 8 \
       -b 48 \
       --fp16 \
       -o \
       -c pretrained/Checkpoint.pth.tar
   

Testing

1. Basic Testing Command

   python tools/su_tracker.py \
       -f exps/SU-T.py \        # Model configuration
       -b 1 \                   # Batch size
       -d 1 \                   # Number of GPUs
       --fp16 \                 # Enable mixed precision
       --fuse \                 # Enable model fusion
       --expn your_exp_name     # Experiment name
   

2. Testing with ReID Module

   python tools/su_tracker.py \
       -f exps/SU-T-ReID.py \   # ReID model configuration
       -b 1 \
       -d 1 \
       --fp16 \
       --fuse \
       --expn your_exp_name
   

Additional Configuration Options

• Model Configuration: Edit exps/SU-T.py or exps/SU-T-ReID.py to modify:

  • Learning rate
  • Training epochs
  • Data augmentation parameters
  • Model architecture settings

• Training Parameters:

  # Additional training options
  --cache        # Cache images in RAM
  --resume       # Resume from a specific checkpoint
  --trt          # Export TensorRT model
  

• Testing Parameters:

  # Additional testing options
  --tsize        # Test image size
  --conf         # Confidence threshold
  --nms          # NMS threshold
  --track_thresh # Tracking threshold
  

📜Tracking Performance

Comparisons on MFT25 dataset

*Note: † indicates the integration of ReID module, Bold indicates the best performance, italics indicate the second-best performance

⁉️Troubleshooting

Common Issues

1. CUDA Out of Memory

   • Reduce batch size
   • Use smaller input resolution
   • Enable mixed precision training

2. Installation Failures

   • Ensure CUDA toolkit matches PyTorch version
   • Try creating environment with pip if conda fails
   • Check system CUDA compatibility

3. Training Issues

   • Verify dataset path and structure
   • Check GPU memory usage
   • Monitor learning rate and loss curves

💕Acknowledgement

A large part of the code is borrowed from ByteTrack, OC_SORT, and HybridSORT. Thanks for their wonderful works!

📖Citation

The citation format will be given after the manuscript is accepted. Using arXiv's citation if needed now.

📑License

This project is released under the MIT License.