coralscapesv2-dinov3-vitb-lora-dpt

DINOv3 ViT-B + LoRA + DPT on CoralscapesV2 (39 classes)

Dense semantic segmentation model with a frozen DINOv3 ViT-B/16 encoder and a DPT decoder (LoRA adapters merged into the backbone), trained on CoralscapesV2 on the 39-class (legacy Coralscapes) label set. Introduced in CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs.

Model Details

• Model type: DINOv3 encoder + DPT decoder (LoRA, merged)
• Finetuned from: DINOv3 ViT-B/16 facebook/dinov3-vitb16-pretrain-lvd1689m
• Label set: 39 classes (model outputs 40 logits, ID 0 = void/ignored)
• Task: dense semantic segmentation of coral-reef imagery

Model Sources

• Training code: dinov3_lora_dpt_coralscapes
• Paper: CoralscapesV2 (OpenReview)
• Project page: josauder.github.io/coralscapesv2
• Dataset: josauder/coralscapesV2

How to Get Started with the Model

This model uses a frozen DINOv3 encoder with a trainable DPT decoder (LoRA merged into the backbone), so it is not a standard transformers.AutoModel. Loading code is shipped with the model as coralscapes_hub_model.py.

The DINOv3 encoder (facebook/dinov3-vitb16-pretrain-lvd1689m) is gated. Accept the license on its model page and authenticate with hf auth login first.

pip install "transformers>=4.44" torch safetensors huggingface_hub pillow datasets

import importlib.util
from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from PIL import Image
from datasets import load_dataset

repo = "EPFL-ECEO/coralscapesv2-dinov3-vitb-lora-dpt"
root = Path(snapshot_download(repo))

# Import the inference class that ships with the model
spec = importlib.util.spec_from_file_location("coralscapes_hub_model", root / "coralscapes_hub_model.py")
chm = importlib.util.module_from_spec(spec)
spec.loader.exec_module(chm)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = chm.Dinov3DPTSegmenter.from_pretrained(root, map_location=device).eval()

# Load an image (or use your own) and predict in a single pass at 768x1376
image = load_dataset("josauder/coralscapesV2", split="test")[0]["image"].convert("RGB")
image = image.resize((1376, 768), Image.BILINEAR)  # (W, H); both divisible by 16
pixel_values = model.processor(images=image, return_tensors="pt", do_resize=False)["pixel_values"].to(device)

with torch.no_grad():
    logits = model(pixel_values)                    # (1, num_classes, H, W)
labels = logits.argmax(1)[0].cpu().numpy()          # HxW array of class IDs

To match the higher-scoring strided 2 x 1024x1024 protocol, run the encoder on the left and right 1024x1024 windows of the native 1024x2048 frame and average the logits (both crops are already divisible by 16). See scripts/verify_hf_hub_model.py / verify_local_checkpoint_patch.py in the training repository for reference implementations of both protocols.

Training Details

Training Data

The model is trained on the CoralscapesV2 dataset, a general-purpose dense semantic segmentation dataset for coral reefs (2433 densely annotated 1024x2048 video frames from 55 reef sites in the Red Sea region).

Training Procedure

The DINOv3 encoder (pretrained on LVD-1689M) is kept frozen and adapted with LoRA (r=16, alpha=32, dropout=0.05), feeding a DPT head (fusion dimension 256, neck hidden sizes [96, 192, 384, 768], reassemble factors [4, 2, 1, 0.5]). Trained with AdamW at learning rate 5e-5, weight decay 0.01, 1000 warmup steps, for a fixed 150 epochs with batch size 8 and mixed precision. Augmentations use multi-scale training (shortest edge 736-1376px across mixed aspect ratios), random resized crop (scale 0.2-1.0, ratio 0.5-2.0), random rotation up to +/-20 degrees, and color jitter. For export, the LoRA weights are merged into the backbone. Evaluation is at 768x1376 (the strided 1024x1024 variant is also reported).

Evaluation

Results on the CoralscapesV2 and Coralscapes (V1) test splits, evaluated over the 39-class label set. single-pass predicts once at 768x1376; strided runs a 1024x1024 window at a 1024px stride (2 windows per frame), matching the protocol of the original Coralscapes paper. Metrics are pixel accuracy and mean IoU (%).

Classes and colors

The model outputs 40 logits (IDs 0-39). ID 0 is void / unlabeled and is ignored during training and evaluation; IDs 1-39 are the legacy Coralscapes (V1) semantic classes. fish is class 9.

The colors below are the official CoralscapesV2 visualization palette. The full id2label and label2color dictionaries are also distributed with the dataset as id2label.json and label2color.json.

<details> <summary><b>Class table (39 classes)</b></summary> </details> <details> <summary><b><code>id2label</code> (id &rarr; class name)</b></summary>

{
  "0": "unlabeled",
  "1": "seagrass",
  "2": "trash",
  "3": "other coral dead",
  "4": "other coral bleached",
  "5": "sand",
  "6": "other coral alive",
  "7": "human",
  "8": "transect tools",
  "9": "fish",
  "10": "algae covered substrate",
  "11": "other animal",
  "12": "unknown hard substrate",
  "13": "background",
  "14": "dark",
  "15": "transect line",
  "16": "massive/meandering bleached",
  "17": "massive/meandering alive",
  "18": "rubble",
  "19": "branching bleached",
  "20": "branching dead",
  "21": "millepora",
  "22": "branching alive",
  "23": "massive/meandering dead",
  "24": "clam",
  "25": "acropora alive",
  "26": "sea cucumber",
  "27": "turbinaria",
  "28": "table acropora alive",
  "29": "sponge",
  "30": "anemone",
  "31": "pocillopora alive",
  "32": "table acropora dead",
  "33": "meandering bleached",
  "34": "stylophora alive",
  "35": "sea urchin",
  "36": "meandering alive",
  "37": "meandering dead",
  "38": "crown of thorn",
  "39": "dead clam"
}

</details> <details> <summary><b><code>label2color</code> (class name &rarr; RGB)</b></summary>

{
  "seagrass": [
    125,
    222,
    125
  ],
  "trash": [
    255,
    0,
    134
  ],
  "other coral dead": [
    114,
    60,
    61
  ],
  "other coral bleached": [
    250,
    224,
    225
  ],
  "sand": [
    194,
    178,
    128
  ],
  "other coral alive": [
    224,
    118,
    119
  ],
  "human": [
    255,
    0,
    0
  ],
  "transect tools": [
    8,
    205,
    12
  ],
  "fish": [
    255,
    255,
    0
  ],
  "algae covered substrate": [
    125,
    163,
    125
  ],
  "other animal": [
    0,
    255,
    255
  ],
  "unknown hard substrate": [
    125,
    125,
    125
  ],
  "background": [
    29,
    162,
    216
  ],
  "dark": [
    31,
    31,
    31
  ],
  "transect line": [
    0,
    255,
    0
  ],
  "massive/meandering bleached": [
    255,
    248,
    228
  ],
  "massive/meandering alive": [
    236,
    150,
    21
  ],
  "rubble": [
    161,
    153,
    128
  ],
  "branching bleached": [
    252,
    231,
    240
  ],
  "branching dead": [
    123,
    50,
    86
  ],
  "millepora": [
    244,
    150,
    115
  ],
  "branching alive": [
    226,
    91,
    157
  ],
  "massive/meandering dead": [
    134,
    86,
    18
  ],
  "clam": [
    189,
    255,
    234
  ],
  "acropora alive": [
    236,
    128,
    255
  ],
  "sea cucumber": [
    0,
    231,
    255
  ],
  "turbinaria": [
    228,
    255,
    119
  ],
  "table acropora alive": [
    189,
    119,
    255
  ],
  "sponge": [
    240,
    80,
    80
  ],
  "anemone": [
    0,
    255,
    189
  ],
  "pocillopora alive": [
    255,
    146,
    150
  ],
  "table acropora dead": [
    85,
    53,
    116
  ],
  "meandering bleached": [
    251,
    243,
    216
  ],
  "stylophora alive": [
    255,
    111,
    194
  ],
  "sea urchin": [
    0,
    142,
    255
  ],
  "meandering alive": [
    230,
    193,
    0
  ],
  "meandering dead": [
    119,
    100,
    14
  ],
  "crown of thorn": [
    179,
    245,
    234
  ],
  "dead clam": [
    89,
    155,
    134
  ]
}

</details>

Citation

If you find this model useful, please cite:

@inproceedings{sauder2026coralscapesv2,
  title={CoralscapesV2: Panoptic and Fine-Grained Visual Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Ruckli, Thomas and Strodomskyt{\.e}, Gabriel{\.e} and Abdallah, Ibrahim Souleiman and Abdi, Rahma Hassan and Awaleh, Djama Goumaneh and Farah, Mohamed Houssein and Nour, Moustapha and Saad, Osama Sharhubil and Altaib, Mustafa Mohammed Khalafallah and Kteifan, Maysoon and Alsoqi, Farah and Zgool, Eyad and Al-Omari, Jafar and Gebreluel, Temesgen Gebremeskel and Abdulkerim, Zekaria Zekeria and Ghirmay, Meron and Beraki, Teklehaimanot and Tuia, Devis and Banc-Prandi, Guilhem},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV) Workshop on Marine Vision},
  year={2026}
}

The dataset builds on the original Coralscapes dataset:

@inproceedings{sauder2025coralscapes,
  title={The Coralscapes Dataset: Semantic Scene Understanding in Coral Reefs},
  author={Sauder, Jonathan and Domazetoski, Viktor and Banc-Prandi, Guilhem and Perna, Gabriela and Meibom, Anders and Tuia, Devis},
  booktitle={Proceedings of the International Conference on Computer Vision (ICCV) Joint Workshop on Marine Vision},
  year={2025}
}

License

Released under the Apache License 2.0. The DINOv3 backbone is subject to the DINOv3 license; the CoralscapesV2 dataset is released under Apache 2.0.