coralscapesv2-dinov3-vitb-lora-dpt-95_class

DINOv3 ViT-B + LoRA + DPT on CoralscapesV2 (95 classes)

Dense semantic segmentation model with a frozen DINOv3 ViT-B/16 encoder and a DPT decoder (LoRA adapters merged into the backbone), trained on CoralscapesV2 on the fine-grained 95-class label set. Introduced in CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs.

Model Details

• Model type: DINOv3 encoder + DPT decoder (LoRA, merged)
• Finetuned from: DINOv3 ViT-B/16 facebook/dinov3-vitb16-pretrain-lvd1689m
• Label set: 95 classes (model outputs 96 logits, ID 0 = void/ignored)
• Task: dense semantic segmentation of coral-reef imagery

Model Sources

• Training code: dinov3_lora_dpt_coralscapes
• Paper: CoralscapesV2 (OpenReview)
• Project page: josauder.github.io/coralscapesv2
• Dataset: josauder/coralscapesV2

How to Get Started with the Model

This model uses a frozen DINOv3 encoder with a trainable DPT decoder (LoRA merged into the backbone), so it is not a standard transformers.AutoModel. Loading code is shipped with the model as coralscapes_hub_model.py.

The DINOv3 encoder (facebook/dinov3-vitb16-pretrain-lvd1689m) is gated. Accept the license on its model page and authenticate with hf auth login first.

pip install "transformers>=4.44" torch safetensors huggingface_hub pillow datasets

import importlib.util
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from PIL import Image
from datasets import load_dataset

repo = "EPFL-ECEO/coralscapesv2-dinov3-vitb-lora-dpt-95_class"
root = Path(snapshot_download(repo))

# Import the inference class that ships with the model
spec = importlib.util.spec_from_file_location("coralscapes_hub_model", root / "coralscapes_hub_model.py")
chm = importlib.util.module_from_spec(spec)
spec.loader.exec_module(chm)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = chm.Dinov3DPTSegmenter.from_pretrained(root, map_location=device).eval()

# Load an image (or use your own) and predict in a single pass at 768x1376
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"].convert("RGB")
image = image.resize((1376, 768), Image.BILINEAR)  # (W, H); both divisible by 16
pixel_values = model.processor(images=image, return_tensors="pt", do_resize=False)["pixel_values"].to(device)

with torch.no_grad():
    logits = model(pixel_values)                    # (1, num_classes, H, W)
labels = logits.argmax(1)[0].cpu().numpy()          # HxW array of class IDs

To match the higher-scoring strided 2 x 1024x1024 protocol, run the encoder on the left and right 1024x1024 windows of the native 1024x2048 frame and average the logits (both crops are already divisible by 16). See scripts/verify_hf_hub_model.py / verify_local_checkpoint_patch.py in the training repository for reference implementations of both protocols.

Training Details

Training Data

The model is trained on the CoralscapesV2 dataset, a general-purpose dense semantic segmentation dataset for coral reefs (2433 densely annotated 1024x2048 video frames from 55 reef sites in the Red Sea region).

Training Procedure

The DINOv3 encoder (pretrained on LVD-1689M) is kept frozen and adapted with LoRA (r=16, alpha=32, dropout=0.05), feeding a DPT head (fusion dimension 256, neck hidden sizes [96, 192, 384, 768], reassemble factors [4, 2, 1, 0.5]). Trained with AdamW at learning rate 5e-5, weight decay 0.01, 1000 warmup steps, for a fixed 150 epochs with batch size 8 and mixed precision. Augmentations use multi-scale training (shortest edge 736-1376px across mixed aspect ratios), random resized crop (scale 0.2-1.0, ratio 0.5-2.0), random rotation up to +/-20 degrees, and color jitter. For export, the LoRA weights are merged into the backbone. Evaluation is at 768x1376 (the strided 1024x1024 variant is also reported).

Evaluation

Results on the CoralscapesV2 and Coralscapes (V1) test splits. This model is trained natively on the 95-class set; we report the native 95-class metrics as well as metrics obtained by mapping the 95-class predictions to the 39-class legacy set for comparison. single-pass predicts once at 768x1376; strided runs a 1024x1024 window at a 1024px stride (2 windows per frame). Metrics are pixel accuracy and mean IoU (%).

Classes and colors

The model outputs 96 logits (IDs 0-95). ID 0 is void / unlabeled and is ignored during training and evaluation; IDs 1-95 are the CoralscapesV2 semantic classes. fish is class 25.

The colors below are the official CoralscapesV2 visualization palette. The full id2label and label2color dictionaries are also distributed with the dataset as id2label.json and label2color.json.

<details> <summary><b>Class table (95 classes)</b></summary> </details> <details> <summary><b><code>id2label</code> (id &rarr; class name)</b></summary>

{
  "0": "unlabeled",
  "1": "acanthaster planci",
  "2": "acropora alive",
  "3": "acropora bleached",
  "4": "acropora dead",
  "5": "algae covered brain coral",
  "6": "algae covered branching coral",
  "7": "algae covered lobophylliidae",
  "8": "algae covered massive coral",
  "9": "algae covered pocillopora",
  "10": "algae covered porites",
  "11": "algae covered substrate",
  "12": "algae covered table acropora",
  "13": "anemone",
  "14": "background",
  "15": "brain coral alive",
  "16": "brain coral bleached",
  "17": "brain coral dead",
  "18": "branching coral alive",
  "19": "branching coral bleached",
  "20": "branching coral dead",
  "21": "cirrhipathes",
  "22": "dark",
  "23": "dead giant clam",
  "24": "feather worm",
  "25": "fish",
  "26": "fungiidae alive",
  "27": "fungiidae bleached",
  "28": "fungiidae dead",
  "29": "galaxea alive",
  "30": "galaxea bleached",
  "31": "galaxea dead",
  "32": "giant clam",
  "33": "goniopora alive",
  "34": "goniopora dead",
  "35": "hard coral alive",
  "36": "hard coral bleached",
  "37": "hard coral dead",
  "38": "hard substrate",
  "39": "human",
  "40": "human-made structure",
  "41": "leather coral alive",
  "42": "lobophylliidae alive",
  "43": "lobophylliidae bleached",
  "44": "lobophylliidae dead",
  "45": "massive alive",
  "46": "massive bleached",
  "47": "massive dead",
  "48": "meandering coral alive",
  "49": "meandering coral bleached",
  "50": "meandering coral dead",
  "51": "millepora",
  "52": "millepora bleached",
  "53": "millepora dead",
  "54": "nephtheidae alive",
  "55": "other animal",
  "56": "other coral alive",
  "57": "other coral bleached",
  "58": "other coral dead",
  "59": "pavona alive",
  "60": "pavona bleached",
  "61": "pavona dead",
  "62": "pocillopora alive",
  "63": "pocillopora bleached",
  "64": "pocillopora dead",
  "65": "porites alive",
  "66": "porites bleached",
  "67": "porites dead",
  "68": "rubble",
  "69": "sand",
  "70": "sea cucumber",
  "71": "sea urchin",
  "72": "seagrass",
  "73": "seriatopora alive",
  "74": "seriatopora bleached",
  "75": "seriatopora dead",
  "76": "soft coral (malacalcyonacea) alive",
  "77": "soft coral (malacalcyonacea) bleached",
  "78": "sponge",
  "79": "stylophora alive",
  "80": "stylophora bleached",
  "81": "stylophora dead",
  "82": "table acropora alive",
  "83": "table acropora bleached",
  "84": "table acropora dead",
  "85": "thin plate/encrusting coral alive",
  "86": "thin plate/encrusting coral bleached",
  "87": "thin plate/encrusting coral dead",
  "88": "transect line",
  "89": "transect tools",
  "90": "trash",
  "91": "turbinaria",
  "92": "turbinaria dead",
  "93": "water column",
  "94": "water surface",
  "95": "xeniidae alive"
}

</details> <details> <summary><b><code>label2color</code> (class name &rarr; RGB)</b></summary>

{
  "acanthaster planci": [
    179,
    245,
    234
  ],
  "acropora alive": [
    236,
    128,
    255
  ],
  "acropora bleached": [
    255,
    236,
    254
  ],
  "acropora dead": [
    118,
    63,
    128
  ],
  "algae covered brain coral": [
    208,
    207,
    58
  ],
  "algae covered branching coral": [
    6,
    71,
    245
  ],
  "algae covered lobophylliidae": [
    50,
    208,
    130
  ],
  "algae covered massive coral": [
    67,
    182,
    208
  ],
  "algae covered pocillopora": [
    225,
    109,
    77
  ],
  "algae covered porites": [
    173,
    81,
    130
  ],
  "algae covered substrate": [
    125,
    163,
    125
  ],
  "algae covered table acropora": [
    48,
    162,
    221
  ],
  "anemone": [
    0,
    255,
    189
  ],
  "background": [
    29,
    162,
    216
  ],
  "brain coral alive": [
    114,
    78,
    120
  ],
  "brain coral bleached": [
    219,
    67,
    238
  ],
  "brain coral dead": [
    78,
    172,
    99
  ],
  "branching coral alive": [
    226,
    91,
    157
  ],
  "branching coral bleached": [
    252,
    231,
    240
  ],
  "branching coral dead": [
    123,
    50,
    86
  ],
  "cirrhipathes": [
    6,
    189,
    22
  ],
  "dark": [
    31,
    31,
    31
  ],
  "dead giant clam": [
    89,
    155,
    134
  ],
  "feather worm": [
    159,
    186,
    38
  ],
  "fish": [
    255,
    255,
    0
  ],
  "fungiidae alive": [
    100,
    133,
    248
  ],
  "fungiidae bleached": [
    200,
    138,
    27
  ],
  "fungiidae dead": [
    3,
    92,
    44
  ],
  "galaxea alive": [
    96,
    188,
    108
  ],
  "galaxea bleached": [
    67,
    243,
    60
  ],
  "galaxea dead": [
    176,
    225,
    11
  ],
  "giant clam": [
    189,
    255,
    234
  ],
  "goniopora alive": [
    181,
    5,
    120
  ],
  "goniopora dead": [
    179,
    191,
    30
  ],
  "hard coral alive": [
    109,
    232,
    188
  ],
  "hard coral bleached": [
    70,
    247,
    149
  ],
  "hard coral dead": [
    114,
    15,
    203
  ],
  "hard substrate": [
    125,
    125,
    125
  ],
  "human": [
    255,
    0,
    0
  ],
  "human-made structure": [
    151,
    171,
    80
  ],
  "leather coral alive": [
    86,
    184,
    94
  ],
  "lobophylliidae alive": [
    201,
    73,
    234
  ],
  "lobophylliidae bleached": [
    253,
    187,
    79
  ],
  "lobophylliidae dead": [
    57,
    106,
    220
  ],
  "massive alive": [
    236,
    150,
    21
  ],
  "massive bleached": [
    255,
    248,
    228
  ],
  "massive dead": [
    134,
    86,
    18
  ],
  "meandering coral alive": [
    230,
    193,
    0
  ],
  "meandering coral bleached": [
    251,
    243,
    216
  ],
  "meandering coral dead": [
    119,
    100,
    14
  ],
  "millepora": [
    244,
    150,
    115
  ],
  "millepora bleached": [
    164,
    226,
    52
  ],
  "millepora dead": [
    119,
    128,
    174
  ],
  "nephtheidae alive": [
    132,
    109,
    91
  ],
  "other animal": [
    0,
    255,
    255
  ],
  "other coral alive": [
    224,
    118,
    119
  ],
  "other coral bleached": [
    250,
    224,
    225
  ],
  "other coral dead": [
    114,
    60,
    61
  ],
  "pavona alive": [
    13,
    191,
    102
  ],
  "pavona bleached": [
    118,
    249,
    130
  ],
  "pavona dead": [
    141,
    130,
    175
  ],
  "pocillopora alive": [
    255,
    146,
    150
  ],
  "pocillopora bleached": [
    250,
    219,
    220
  ],
  "pocillopora dead": [
    130,
    66,
    66
  ],
  "porites alive": [
    255,
    209,
    71
  ],
  "porites bleached": [
    213,
    10,
    25
  ],
  "porites dead": [
    40,
    24,
    243
  ],
  "rubble": [
    161,
    153,
    128
  ],
  "sand": [
    194,
    178,
    128
  ],
  "sea cucumber": [
    0,
    231,
    255
  ],
  "sea urchin": [
    0,
    142,
    255
  ],
  "seagrass": [
    125,
    222,
    125
  ],
  "seriatopora alive": [
    191,
    19,
    6
  ],
  "seriatopora bleached": [
    247,
    89,
    191
  ],
  "seriatopora dead": [
    75,
    131,
    128
  ],
  "soft coral (malacalcyonacea) alive": [
    147,
    144,
    54
  ],
  "soft coral (malacalcyonacea) bleached": [
    7,
    227,
    247
  ],
  "sponge": [
    240,
    80,
    80
  ],
  "stylophora alive": [
    255,
    111,
    194
  ],
  "stylophora bleached": [
    255,
    220,
    241
  ],
  "stylophora dead": [
    123,
    54,
    94
  ],
  "table acropora alive": [
    189,
    119,
    255
  ],
  "table acropora bleached": [
    244,
    233,
    253
  ],
  "table acropora dead": [
    85,
    53,
    116
  ],
  "thin plate/encrusting coral alive": [
    91,
    28,
    190
  ],
  "thin plate/encrusting coral bleached": [
    42,
    209,
    57
  ],
  "thin plate/encrusting coral dead": [
    8,
    19,
    242
  ],
  "transect line": [
    0,
    255,
    0
  ],
  "transect tools": [
    8,
    205,
    12
  ],
  "trash": [
    255,
    0,
    134
  ],
  "turbinaria": [
    228,
    255,
    119
  ],
  "turbinaria dead": [
    169,
    30,
    70
  ],
  "water column": [
    31,
    0,
    166
  ],
  "water surface": [
    0,
    255,
    248
  ],
  "xeniidae alive": [
    139,
    142,
    168
  ]
}

</details>

Citation

If you find this model useful, please cite:

@inproceedings{sauder2026coralscapesv2,
  title={CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Ruckli, Thomas and Strodomskyt{\.e}, Gabriel{\.e} and Abdallah, Ibrahim Souleiman and Abdi, Rahma Hassan and Awaleh, Djama Goumaneh and Farah, Mohamed Houssein and Nour, Moustapha and Saad, Osama Sharhubil and Altaib, Mustafa Mohammed Khalafallah and Kteifan, Maysoon and Alsoqi, Farah and Zgool, Eyad and Al-Omari, Jafar and Gebreluel, Temesgen Gebremeskel and Abdulkerim, Zekaria Zekeria and Ghirmay, Meron and Beraki, Teklehaimanot and Tuia, Devis and Banc-Prandi, Guilhem},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV) Workshop on Marine Vision},
  year={2026}
}

The dataset builds on the original Coralscapes dataset:

@inproceedings{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Domazetoski, Viktor and Banc-Prandi, Guilhem and Perna, Gabriela and Meibom, Anders and Tuia, Devis},
  booktitle={Proceedings of the International Conference on Computer Vision (ICCV) Joint Workshop on Marine Vision},
  year={2025}
}

License

Released under the Apache License 2.0. The DINOv3 backbone is subject to the DINOv3 license; the CoralscapesV2 dataset is released under Apache 2.0.