Downloads · 30 days
0
nilsho01/unet-resnet34-vhr-buildings
unet-resnet34-vhr-buildings is a image segmentation model from nilsho01. Use it for the image segmentation task on the model card, and read the license before you ship it in a product. The card lists the license as agpl-3.0.
The problem is that the current model (01.08.2026) in fAIr's UNET pipeline is a tree crown model which does not really fit to the purpose of finding buidlings. Certainly, it is well enough for fine-tuning downstream b…
Downloads · 30 days
0
Access
Public
Updated Aug 5, 2026
Repo size
199 MB
Likes
0
Public
Click a slice to open those files.
.pth196 MB · 98%
From the Hugging Face model README
The problem is that the current model (01.08.2026) in fAIr's UNET pipeline is a tree crown model which does not really fit to the purpose of finding buidlings. Certainly, it is well enough for fine-tuning downstream but this could require more computational ressources, than simply providing a baseline which is capable of building segmentation across multiple regions. This hypothesis will be tested along other findings in this repository as well. The motivation behind the model card is to provide a model which was fine tuned on hotosm/vhr-building-segmentation to have a baseline for a ressource efficient and more accurate fine-tuning for building segmentation. Furthermore, it was noted that the UNET does not have the same mask, boundary and distance heads as the vision transformer, which is used in the same repository. This makes it hard to compare them downstream and choose the most suitable model considering computation ressources and accuracy for fine-tuning. The current UNET implementation only segments the image but does not provide the information to apply a watershed algorithm. Hence, as a second motivation a UNET with two extra output maps similar to the ViT (dinov3s_buildings) was trained with the goal of having a UNET with the same output capabilities like the ViT.
Based on these two motivations the repository contains two kinds of models:
unet_bldg_base.pth — drop-in replacement base for fAIr's UNet
pipeline (2 output channels, loads into the existing pipeline
unchanged). This is the file the fAIr pull request is about.unet_bldg_instance.pth — same model with two extra output heads
(boundary, signed distance); a watershed on its outputs separates
touching buildings into individual footprints. Standalone use only —
the current fAIr pipeline does not consume the extra heads.Both checkpoints were trained on the full, unmodified hotosm/vhr-building-segmentation dataset (OpenAerialMap imagery with OpenStreetMap labels, mostly Global South regions). The validation split drove early stopping and checkpoint selection; the test split was never touched during training.
The base follows fAIr's production training recipe so that it behaves like the pipeline expects; the instance model uses a stronger recipe since it is meant for standalone use:
| Parameter | unet_bldg_base.pth | unet_bldg_instance.pth |
|---|---|---|
| Train / val / test size | 57,890 / 7,237 / 7,236 chips | same |
| Input | RGB 256x256, scaled by 1/255 | same |
| Architecture | UNet, ResNet34 encoder (segmentation_models_pytorch), 2-channel head | same encoder/decoder, 3-channel head (mask, boundary, signed distance) |
| Initialisation | ImageNet encoder, random decoder | same |
| Trained parameters | decoder + head (3.2M); encoder frozen | all (~24.4M); encoder LR 1e-4, decoder/head LR 1e-3 |
| Learning rate | 1e-3 | encoder 1e-4, decoder/heads 1e-3 |
| Optimizer | AdamW, weight decay 1e-4 | same |
| Loss | 2-class cross-entropy (fAIr's production loss) | Generalized Cross-Entropy (q=0.7) + Tversky (alpha=0.3, beta=0.7) on the mask, + boundary BCE (weight 0.27) + signed-distance MSE on tanh (weight 0.47) |
| Batch size | 32 | 32 |
| Augmentation | none (production preprocessing) | horizontal/vertical flips, rotation +-15 deg, zoom-crop (scale 0.7-1.0), color jitter (brightness/contrast 0.25, saturation 0.15) |
| LR schedule | cosine annealing (T_max 50) | same |
| Weight averaging | none | EMA, decay 0.99 (validation and checkpoint use EMA weights) |
| Gradient clipping | max-norm 1.0 | same |
| Precision | mixed fp16 | same |
| Early stopping | patience 5 on validation CE; stopped at epoch 9 of max 50 (~0.6 h) | patience 7 on validation loss; stopped at epoch 23 of max 50 (~2.3 h) |
| GPU | RTX 2080 Ti (single) | same |
| Seed | 1337 (single run per checkpoint) | same |
Training script:
phase2_pretrain_cnn_base.py
(fork branch; latest options being upstreamed).
First, the unet_bldg_base (base) and unet_bldg_instance (instance) model were compared to the ViT regarding their pixel f1 score. This was calculated with:
F1 = 2TP/ (2TP + FP + FN)
This is a pixel wise score calculated for the whole image and if one imagines an image as a two class task: building or background it focuses on the building class only. This is because the majority in the images is background which would inflate the score otherwise.
Given the binary character of the prediction the threshold for deciding such was set to 0.5 for the instance model and the ViT, where this is close to the best performing operating point; the base is not calibrated at 0.5 (see the regions table below).
In the following the base, instance, ViT and tree crown model are compared regarding their capability of segmenting buildings on clips based on the f1 score for the validation and test dataset in hotosm/vhr-building-segmentation. In the table below one can see that the instance and ViT model show similar or the same performance on these datasets highlighting that it can be meaningful to fine-tune a UNET, if computational ressources are limited, since the UNET can be significantly smaller and thus inference or fine-tuning will be faster. The base is not calibrated at threshold 0.5 which explains the lower zero-shot numbers; its purpose is the fine-tuning warm start inside the fAIr pipeline (see below), where it is the strongest UNET base. This can be even more optimized if smaller encoders are used such as resnet-18 or tu-resnet10t. Finally, it can be seen that the tree crown model underperforms, since it was trained on a different task than finding buildings.
| Model | Pixel F1 (val / test) | Size |
|---|---|---|
unet_bldg_base.pth (this repo) | 40.2 / 50.9 | ~94 MB |
unet_bldg_instance.pth (this repo) | 74.3 / 61.6 | ~94 MB |
| shipped DINOv3-ViT flagship | 72.9 / 59.9 | ~462 MB |
| previous UNet base (tree-crown) | 0.0 / 0.0 | — |
This repository assumes that in real-world scenarios the users would apply the models to certain regions with different characteristics such as building density or different geographies. To simulate this behaviour 6 regions were chosen: Banepa, Jakarta, Accra, Bogota, Manila and Nairobi. The Banepa chips are OpenAerialMap drone imagery with OpenStreetMap labels (fAIr's sample region); the other five regions were downloaded as ArcGIS World Imagery tiles (zoom 18) with OpenStreetMap building labels — a different imagery source than the training data. For each region 100 clips were available (120 for Banepa). Then, the zero-shot behaviour of the 3 models: base, instance and ViT were compared for the different regions for a fixed threshold at 0.5 and the best operating threshold. In comparison to the previous evaluation the best operating threshold can vary significantly, since the clips from the 6 regions are out of the domain of the training data: the footprint of every region chip was checked against all 72,363 chips in hotosm/vhr-building-segmentation (train, validation and test) via bounding-box intersection and none overlap. The training split also contains no chips from Nepal or Indonesia at all.
In general, one can see that the ViT shows the best performs on the different regions. The instance and base model behave similarly for the best operating threshold, where the base needs to be calibrated depending on the region because it shows a significantly lower performance if the threshold is fixed at 0.5. This is not the case for the instance model and marginally for the ViT indicating that if both are implemented in production the user may not need to choose the operating point which could icnrease performance and user adoption.
A closer look reveils that the performance from region to region varies greatly. For instance, all three models underperform on Bogota with F1 ~ 25 while they show high performance on Banepa with F1~73. There are multiple reasons for this discrepancy including the fact that Bogota might be an example for out-of-domain chips. However, when manually going through the chips from Bogota one could see that many buildings were missing labels which is part of the reason why certain scores are deflated. Two examples with one for Bogota (top) and one for Banepa (bottom) are provided below the table where the original image is shown on the left, the image with the labels in the center and the prediction on the right.
| Region | base ([email protected] / F1@best / R@best) | instance model ([email protected] / F1@best / R@best) | shipped DINOv3-ViT ([email protected] / F1@best / R@best) |
|---|---|---|---|
| Banepa | 48.4 / 67.6 / 75.6 | 73.5 / 73.9 / 78.9 | 65.9 / 73.3 / 79.6 |
| Jakarta | 49.2 / 63.1 / 81.0 | 61.5 / 62.2 / 69.1 | 64.1 / 66.0 / 79.9 |
| Accra | 34.1 / 50.6 / 57.9 | 44.5 / 45.9 / 39.6 | 59.5 / 59.9 / 72.4 |
| Bogota | 26.3 / 27.5 / 38.4 | 21.8 / 22.7 / 67.9 | 23.1 / 29.9 / 45.9 |
| Manila | 56.5 / 70.4 / 82.9 | 70.0 / 70.6 / 72.7 | 73.7 / 75.1 / 84.7 |
| Nairobi | 52.2 / 56.7 / 68.4 | 59.3 / 59.3 / 69.7 | 61.1 / 61.3 / 70.6 |
| pooled | 44.5 / 56.0 / 67.4 | 55.1 / 55.8 / 66.3 | 57.9 / 60.9 / 72.2 |
R@best is the building-pixel recall at the same best operating threshold as the reported F1@best. A high recall together with a low F1 (Bogota: instance model recall 67.9, F1 22.7) means the model finds the buildings but the labels do not contain them — missing labels, not missed buildings. The reverse pattern is a genuine model gap (Accra: instance model recall 39.6 while the ViT reaches 72.4 on the same chips).
Examples of clips from Bogota
In Bogota entire dense blocks carry no OSM labels at all — 15 of the 100 evaluation chips have a completely empty label while the model predicts substantial building area on them. The model predicts the buildings actually fairly well (on the right):

In Banepa the labels sit on the rooftops and the prediction agrees (chip F1 90.0 and 88.8); only 1 of 120 chips has an empty label, and on that chip the model predicts no buildings either:

The previous chapter analyzed how the models perform in a zero-shot manner on novel and distinct regions. fAIr's goal is to provide a pipeline to fine-tune models on regions because, as shown before, accuracies can differ even under consideration of wrong labelling. To simulate this, each candidate warm start was fine-tuned with five-fold cross-validation: the region's 100–120 chips were partitioned into five disjoint test folds (~20 chips each); within each fold the remaining chips were split 80/20 into a training pool and validation set, and 32 training chips were drawn from the pool. The same analysis was conducted for 8 and 16 chips, where similar tendencies were noticed; 32 chips was deemed realistic considering the real-world application scenario. The model's best confidence threshold was chosen on the validation set and the test fold was scored once at that threshold. The reported standard deviation is over the five folds. Fine-tuning used fAIr's production recipe (frozen encoder, cross-entropy, lr 1e-4), extended with early stopping (patience 5, max 30 epochs).
The table below shows the performance for the tree-crown model, the same model but with a random initialized decoder, the base model (newly trained) and the ViT. One can see that the tree crown model and the randomly initialized model nearly perform in the same fashion highlighting that the current baseline model does not possess knowledge about buildings beyond the variance of the given train split. Comparing this with the new base and the ViT, performance can be significantly improved across all 6 regions, with pooled f1 scores 7–10 points higher (+9.3 for the base and +10.4 for the ViT over the tree-crown base, +6.9 for the base over the random initialization). Furthermore, the tree crown model shows a high variance for Banepa: as the figure below shows, some of its fine-tuning runs never recover from the tree-crown initialization within the epoch budget — even though Banepa has the cleanest labels of the six regions. Here, the new trained UNET or the ViT show more robust behaviour.
Cells: test F1 ± standard deviation over the 5 folds (R = mean test recall at the val-selected threshold):
| Fine-tuning warm start | Banepa | Jakarta | Accra | Bogota | Manila | Nairobi | pooled |
|---|---|---|---|---|---|---|---|
| tree-crown base (previously shipped) | 51.2 ± 21.9 (R 62.1) | 62.6 ± 1.6 (R 78.8) | 48.6 ± 3.4 (R 68.7) | 17.7 ± 8.4 (R 36.5) | 67.5 ± 1.6 (R 82.2) | 52.3 ± 0.9 (R 66.6) | 50.0 (R 65.8) |
| random decoder init (no base) | 66.8 ± 2.0 (R 72.9) | 62.1 ± 1.2 (R 78.1) | 46.9 ± 2.3 (R 67.7) | 20.3 ± 7.0 (R 32.0) | 67.0 ± 1.3 (R 85.0) | 51.1 ± 1.4 (R 62.6) | 52.4 (R 66.4) |
unet_bldg_base.pth | 74.2 ± 3.4 (R 81.0) | 63.3 ± 0.9 (R 76.6) | 54.4 ± 2.8 (R 68.7) | 34.5 ± 7.4 (R 44.2) | 68.9 ± 4.6 (R 76.4) | 60.4 ± 2.2 (R 68.6) | 59.3 (R 69.3) |
| shipped DINOv3-ViT base | 72.3 ± 4.3 (R 76.8) | 65.7 ± 0.8 (R 80.4) | 58.2 ± 3.4 (R 69.2) | 29.7 ± 4.1 (R 35.7) | 74.8 ± 1.6 (R 85.7) | 61.6 ± 1.6 (R 69.1) | 60.4 (R 69.5) |
All in all, it is highly recommended to replace the tree-crown base with the unet_bldg_base checkpoint in future implementations. As a fine-tuning warm start the checkpoint outperforms both the tree-crown base and a random decoder initialization in all 6 regions across 3 continents. Furthermore, it avoids the catastrophic Banepa failure of the tree-crown base (± 3.4 vs ± 21.9) and reaches a higher recall when fine-tuning (69.3 vs 65.8 pooled).
When floods or similar occur fine tuning is likely necessary given the different character of the background in comparison to the images in the training dataset (hotosm/vhr-buildings-segmentation). However, the models might also be just used for labelling images of rural settlements or similar. The question is whether fine-tuning such on a subset is necessary then or if they ViT and unet work in a zero shot manner anyways. The table above and the the figure below provide an indication to this problem. In the figure one can see how a green line (ViT), a red line (tree-crown base) and a blue line (base unet) as a representation of many of the fine-tuning campaigns. Here, the ViT is stagnating which is why the checkpoint is saved in the first epoch and also matches the observation that in the zero shot table and the fine-tuning campaign table the results for the corresponding regions do almost not differ. These results suggest that a single global model may be sufficient for most urban regions: zero-shot recall is high across all six regions, fine-tuning added nothing in Jakarta and Manila, and a large share of the remaining F1 gap is explained by label defects rather than model failures (Bogota: recall 67.9 vs F1 22.7). Under this view fine-tuning would only be necessary for true edge cases such as flooded areas or unusual building typologies — provided a cheap test detects them. So if the goal is to maximize accuracy and minimze user friction the solution would be to only offer the ViT and finding out how to find truly out-of-domain scenarios from a few clips to then fine-tune the vision transformer. If computational ressources are limited, the presented unet can be deployed as a default version but should be fine-tuned using a generous epoch window with early stopping.
A few things are still open. All six regions are cities, so it is not known yet how the models behave in real edge cases like flooded areas or remote villages. It has also not been shown that cleaning the training data actually makes the zero-shot model better — in the experiments conducted here, models trained on cleaned subsets were no better than models trained on random subsets of the same size. And Accra is a reminder that not every problem is a label problem: there the UNET simply misses many buildings that the ViT finds, and no amount of label cleaning will fix that.

The informed user might think that the f1 scores are relatively low given the specific task definition of finding buildings, and the capapbilities of a UNET or ViT in comparable tasks. As discussed before this is most likely because train, validation and test data are labelled inaccurately which can be separated in three kinds: missing labels, phantom labels and shifted labels where one example is provided for each category below representatively from the dataset hotosm/vhr-buildings-segmentation. This means that the actual reported scores could be higher on each of the datasets or that the models can still be significantly improved if models are trained on correctly labelled images. This is interesting for future research where models trained on clean data could be made more efficient and thus faster, smaller requiring less computational ressources since they need to represent less noise.
Buildings with no label at all — the label is empty; the building is plainly there (right: model detection, confirmed by inspection):

Labels with no buildings — building rectangles on bare ground; the human-corrected label is empty:

Offset labels — same buildings, label displaced off the rooftops (label-vs-correction overlap only 9%; correction by human review):

The base and instance checkpoints share the same trained ResNet34-UNet. The instance
variant adds boundary and signed-distance output heads; a watershed on
mask + distance splits touching buildings into separate instances (the
same post-processing as fAIr's DINOv3 stack:
dinov3_hot.infer.instance_separate).
Examples from the held-out test split — left: input image; middle:
unet_bldg_base.pth output (semantic mask — touching buildings merge
into one region); right: unet_bldg_instance.pth + watershed, each
separated building instance in its own color. Dense, medium and sparse
scenes:

import torch
import segmentation_models_pytorch as smp
# base model (drop-in for the fAIr pipeline, 2 channels)
model = smp.create_model("unet", encoder_name="resnet34", encoder_weights=None,
classes=2, in_channels=3)
state = torch.hub.load_state_dict_from_url(
"https://hf.co/nilsho01/unet-resnet34-vhr-buildings/resolve/main/unet_bldg_base.pth",
map_location="cpu", weights_only=True)
model.load_state_dict(state)
x = torch.rand(1, 3, 256, 256) # RGB / 255
mask = model(x).argmax(1) # argmax; out of domain a lower threshold gains several F1 (see regions table)
# instance model (standalone, 3 channels)
inst = smp.create_model("unet", encoder_name="resnet34", encoder_weights=None,
classes=3, in_channels=3)
state = torch.hub.load_state_dict_from_url(
"https://hf.co/nilsho01/unet-resnet34-vhr-buildings/resolve/main/unet_bldg_instance.pth",
map_location="cpu", weights_only=True)
inst.load_state_dict(state)
out = inst(x) # (1, 3, 256, 256)
mask_prob = torch.sigmoid(out[:, 0])
distance = torch.tanh(out[:, 2])
# from dinov3_hot.infer import instance_separate
# labels = instance_separate(mask_prob[0].numpy(), distance[0].numpy(),
# mask_threshold=0.5, seed_min_distance=6,
# large_blob_area_px=1500, h_maxima_depth=0.2)