Shark-Patrol-yolo

Aerial Shark–Swimmer YOLO v2

Two YOLOv8s detectors for overhead (drone / beach-tower) imagery. Classes:

License: personal / non-commercial only — see LICENSE.
Not a safety system. Do not use as the sole input for beach closures, rescue, or maritime alerting.

Related earlier single-class beta: True2456/aerial-shark-yolo-beta.

---

Goal

Collecting enough real labeled aerial shark footage is slow and expensive. The working hypothesis here is the same one used in sim-to-real robotics and synthetic detection work: generate controlled synthetic overhead scenes with a world model (NVIDIA Cosmos 3), auto-label them with an open-vocabulary detector, mix in real shark stills, then actively mine hard negatives (objects that look like sharks from altitude — surfboards, dolphins, boat wakes) so the classifier stops treating them as positives.

This release is the practical result of that loop for a two-class (shark / swimmer) beach-surveillance detector. Both published checkpoints were trained with synthetic Cosmos 3 shark scenes + 28 real shark photos (plus hard negatives).

---

Workflow

NVIDIA Cosmos 3 synthetic aerial ──┐
                                   ├─► Grounding DINO auto-boxes ─► YOLO labels
real shark stills (irlshark) ──────┘         │
                                             ▼
                                  quality filters (score, area, NMS-ish caps)
                                             │
hard negatives (boats, dolphins,             ▼
surfers, empty ocean) ── empty .txt ─► Ultralytics YOLOv8s train (MPS)
                                             │
                                      Combined v2 weights
                                             │
                                 + surfer stills (empty labels)
                                 low LR AdamW fine-tune
                                             ▼
                                   surfneg fine-tuned weights

1. Synthetic generation (NVIDIA Cosmos 3)

Synthetic overhead beach / ocean scenes were generated with NVIDIA Cosmos 3 (local Cosmos3-Nano MLX build, driven through the SurfLifeGen / MLX_TrainDetect tooling). Cosmos is NVIDIA’s physical-AI world foundation model family, used here as a generator for aerial water scenes with sharks and related clutter — including variation in water appearance (turbidity, sun glint, surface texture) and shark silhouette / submersion cues that matter from altitude (NVIDIA Cosmos; Cosmos 3 overview).

Generated frames were ranked into high / med / low quality buckets; only high + med entered training. Prompt / viewpoint / altitude diversity is the domain-randomization side of the same loop (Tobin et al., IROS 2017).

2. Auto-labeling with Grounding DINO

Boxes for synthetic frames and for the real shark stills were proposed by Grounding DINO (IDEA-Research/grounding-dino-base) with text prompts such as:

submerged shark . shark in water . marine predator silhouette .

Grounding DINO is an open-set detector that grounds language queries to boxes (Liu et al., “Grounding DINO…”, arXiv:2303.05499; ECCV 2024). Export to YOLO class cx cy w h (normalized) was done after:

• score thresholds (stricter for swimmer prompts, which overfire),
• min/max box area relative to the image,
• keeping only the top-N boxes per image.

Manual correction passes exist on parts of the aerial set (labels_corrected/, correction reports). Labels are still imperfect; treat them as research-grade.

3. Real shark photos — yes, both models use them

Both Combined v2 and the surfneg fine-tune include real shark-positive images, not synthetic-only.

The 28 irlshark frames are real aerial / drone screenshots of sharks, auto-boxed with Grounding DINO then exported to YOLO. They are the labeled real-shark bridge; synthetic Cosmos 3 frames supply volume and lighting / water diversity. Without the IRL positives, earlier synthetic-only runs looked strong on synthetic val and weaker on beach video.

4. Hard negatives

False positives on surfers, boats, and dolphins are the failure mode that matters in beach video. Those photos were added with empty label files (Ultralytics treats them as background images). That is classical hard-negative mining / bootstrapping: show the network the mistakes you care about and force the classification head to score them as “no object” (Shrivastava et al., OHEM, CVPR 2016; long lineage back to discriminative part models and SVM hard-neg mining).

Combined v2 already included boats, dolphins, real swimmers-without-boxes, and empty aerial frames. The surfneg checkpoint adds 24 real surfer photos and fine-tunes from v2 rather than training from yolov8s.pt again (from-scratch retrain with the same negs underperformed on video).

5. Detector training

Architecture: YOLOv8s via Ultralytics (COCO-pretrained). Training was on Apple Silicon MPS.

Held-out val (Ultralytics default split metrics at best checkpoint):

Swimmer metrics are noisy (few val instances). Prefer surfneg on real beach video; keep combined v2 if you need a checkpoint that never saw the surfer stills.

---

Quick start

pip install ultralytics

from ultralytics import YOLO
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="True2456/Shark-Patrol-yolo",
    filename="weights/yolov8s_aerial_shark_swimmer_surfneg.pt",
)
model = YOLO(path)
model.predict(source="beach.mp4", device="mps", conf=0.25, show=True)

Or download both .pt files from the Files tab and load by path.

Useful runtime knobs for video:

• Raise conf (e.g. 0.35–0.45) if surfers still trigger low-confidence shark boxes.
• Class filter: only show sharks with classes=[0].
• These models are frame detectors, not trackers; for temporal stability wrap with ByteTrack / BoT-SORT via Ultralytics track().

---

Data mix (Combined v2)

Approximate image counts by source (from training manifest):

Surfneg keeps the same shark positives (including all 28 real shark stills) and adds 24 labeled-empty surfer stills.

---

Limitations

• Auto-labels from Grounding DINO are not gold-standard boxes; expect missed fins and occasional over-tight/over-loose rectangles.
• Real shark positives are still a small set (28 frames). Cosmos 3 covers a wide range of water turbidity, sun glint, and silhouette appearance in synthetic form, but more labeled real shark footage (especially from the target camera / altitude) remains the main way to close remaining video-specific gaps.
• Surfer suppression is only as good as the negative set. Still FPs on a given drone clip → crop those frames, empty-label them, fine-tune again from the surfneg weights (same recipe, lower LR).
• Val sets are small; mAP numbers are directional, not a substitute for watching the target video.
• Two-class head: boats/dolphins are not output classes — they are backgrounds. If you need boat/dolphin IDs, label them explicitly and retrain with nc > 2.

---

References

1. NVIDIA — Cosmos (world foundation models for physical AI / synthetic data). https://github.com/NVIDIA/Cosmos · https://research.nvidia.com/labs/cosmos-lab/cosmos3/
2. Jocher, G., Chaurasia, A., Qiu, J. — Ultralytics YOLO (YOLOv8). https://github.com/ultralytics/ultralytics
3. Liu, S. et al. — Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv:2303.05499, 2023 / ECCV 2024. https://arxiv.org/abs/2303.05499
4. Tobin, J. et al. — Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. IROS 2017. https://arxiv.org/abs/1703.06907
5. Shrivastava, A., Gupta, A., Girshick, R. — Training Region-based Object Detectors with Online Hard Example Mining. CVPR 2016. https://arxiv.org/abs/1604.03540
6. Yosinski, J. et al. — How transferable are features in deep neural networks? NeurIPS 2014. https://arxiv.org/abs/1411.1792

---

Citation

@misc{true2456_aerial_shark_swimmer_yolo_v2,
  author       = {True2456},
  title        = {Aerial Shark–Swimmer YOLO v2},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/True2456/Shark-Patrol-yolo}},
  note         = {Personal use only. Experimental detector weights.}
}