bambi-species-classification

BAMBI Species Classifiers

Three heads that classify red deer, roe deer and wild boar in nadir UAV imagery from frozen DINOv3 ViT-H+ features: one for thermal, one for RGB, and one for the two together.

They are the species counterpart to cpraschl/bambi-occlusion-classifiers and share its interface exactly, so code that consumes one can consume the other by swapping the file. Both are built for the BAMBI dataset.

Files

Each holds species_thermal.pt (1280-d in), species_rgb.pt (1280-d) and species_matched.pt (2560-d). The projections are not interchangeable: a head expects the feature distribution of the projection it was trained on. No ALFS variant is included.

Usage

The models consume raw DINOv3 ViT-H+ CLS features, not images.

import torch

m = torch.jit.load("non_geo/species_matched.pt")
emb, probs = m(feats)          # feats: (N, 2560) float32, [RGB | thermal]
# emb   : (N, 256)  L2-normalised embedding
# probs : (N, 3)    in the order given by m.classes
print(m.classes)               # ['red_deer', 'roe_deer', 'wild_boar']

Exported attributes: classes, emb_dim (256), in_dim, modality, projection, reduction.

To produce the features, crop the animal square around the box centre at 1.5x the longer side, resize to 224, normalise with ImageNet statistics, and take the CLS token of DINOv3 ViT-H+. Standardisation of the features is inside the model; feed the raw CLS tokens.

Architecture

features -> standardize(mu, sigma) -> Linear(D, 512) -> ReLU
         -> Linear(512, 256) -> L2-normalize -> linear classifier -> softmax

The projection is trained with batch-hard triplet loss on L2 distance; the classifier is then fitted on the frozen 256-d embedding. Batch norm is folded into the preceding linear layer after training, so inference is deterministic without requiring .eval(). The matched model concatenates the two feature vectors before the projection, so the head learns a joint representation rather than combining two separate decisions.

This follows Markoff et al., When One Modality Is Not Enough (arXiv:2608.06973), which used the same construction for sex and life-stage on red deer.

Results

5-fold cross-validation, grouped by flight, on key frames that the matching occlusion head calls clear.

non_geo (perspective): 36,091 boxes, 2,838 tracks, 244 flights.

geo_1k (orthographic, 1024): 31,730 boxes, 2,792 tracks, 230 flights.

geo_2k (orthographic, 2048): 32,899 boxes, 2,667 tracks, 231 flights.

Note the per-track column: geo_2k reaches 0.909, above non_geo's 0.889. Per frame the perspective heads stay ahead, but once a track is resolved by a majority vote over its frames, the orthographic 2k head is the strongest of the three. For geo-referenced tracking, which works in projected space anyway, that is the number that matters.

Fusion beats either sensor alone in perspective (+2.3 points over RGB); in the orthographic view it adds almost nothing (+0.4). Thermal is the weaker modality throughout, which is expected -- species identity lives in texture and outline, and thermal renders an animal as a warm blob.

Confusion for matched (rows = truth, columns = predicted):

alfs_1k / alfs_2k (light-field integration, unfiltered): 61,516 and 61,536 boxes, 3,772 tracks, 244 flights.

These are the weakest heads per frame, and the reason is inherent to the method rather than fixable by rendering harder. The synthetic aperture spans ±45 frames at stride 3, roughly three seconds. Static ground sharpens and canopy blurs away, which is the point -- but a moving animal smears across its own track, and species identity lives in the outline and texture the smear destroys. Doubling the render buys +1.3 points, about what it buys the orthographic arm, so resolution is not what limits them.

Where the ALFS heads are the right choice

On frames a perspective occlusion head calls occluded -- the material light-field integration exists for -- the ordering changes:

alfs_1k has the best per-track accuracy of any head on this subset, above perspective. The same occlusion head also flags only 37-38% of ALFS frames as occluded, against 45% of perspective and 50% of orthographic ones, so the integration measurably reduces apparent occlusion rather than merely being robust to it.

So: perspective for general use, ALFS for canopy-heavy material where the animal is intermittently hidden and a track-level answer is what is wanted.

With and without occlusion filtering

Every table above filters frames through the matching head of bambi-occlusion-classifiers and keeps only those it calls clear, which is roughly half of them. Without that step the same heads, trained and evaluated on every frame, score materially lower per frame:

Filtering is worth +8.8 points to the perspective arm. It is worth almost nothing to ALFS -- 1.2 points down at 1k, 2.4 up at 2k, both inside the fold spread. That is why the ALFS heads are released trained on unfiltered frames: the filter is not part of their intended pipeline, and light-field integration has already done the job it would do.

Per modality, unfiltered:

Filtering is worth most to the perspective arm (+8.8 points) and least to geo_1k (+1.5), and it costs thermal the most to go without -- an occluded animal in thermal is a partial warm blob, which is exactly the case the modality has least margin for.

The per-track picture is different, and worth knowing before switching the filter on. Majority voting over a track benefits from having more frames to vote with, so filtering can cost per-track accuracy even while it improves per-frame accuracy:

So filter when labelling individual frames; for track-level decisions on geo_1k, keeping every frame is the better choice.

Why the orthographic heads score lower

Partly resolution, and partly the projection itself. A 70 m ortho window over 1024 px is 6.8 cm/px, which leaves the animal smaller than in the source frame; doubling to 2048 more than reverses that:

Rendering at 2048 recovers about half the deficit -- matched balanced accuracy 0.809 -> 0.854 against perspective's 0.904 -- and roe deer, the smallest and weakest class, recovers most: 63% correct at 1k, 77% at 2k, against 87% in perspective. So undersampling was real and worth fixing.

But it is not the whole story. At 2048 the ortho crops are larger in pixels than the perspective ones and still score five points lower. Those extra pixels are resampled from the same source frames, so they carry no new detail: the render can only redistribute the sampling the sensor already made, while adding interpolation blur and whatever error the DEM and pose carry. Rendering beyond 2048 should not be expected to close the remaining gap.

Read the split protocol before comparing numbers

97.4% of BAMBI flights contain exactly one of the three species, so flight identity is very nearly a perfect proxy for the label. Splitting by track leaves other tracks from the same flight in training, and a head can then score well by recognising the session -- its terrain, illumination and thermal calibration -- without learning anything about animals:

The flight-grouped number is the one that means something. Track grouping is appropriate for the paper's sex and life-stage tasks, where a single flight contains both classes; it is not appropriate for species.

Interpolated labels were not used

BAMBI is annotated on key frames. Densifying with mot_interpolation.py (step 5, gaps capped at 30 frames) more than doubles the data and makes results worse under flight grouping:

Under track grouping the two are indistinguishable (0.967 vs 0.969), which is the tell: interpolated boxes are near-duplicates that inflate a track-grouped score while adding drift-induced label noise that only shows up on unseen flights. These models are trained on key frames only.

Training data

266 flights of the BAMBI base release containing the three species: 1,605 red deer, 646 roe deer and 1,810 wild boar tracks. Species labels come from each flight's own <flight>_gt.txt, which carries the species name in plain text.

Thermal boxes are the released annotations. RGB boxes are not -- the two modalities are not spatially aligned, so each thermal box was transferred onto the RGB frame by detecting the animals with OWL and matching detections to box centres with the Hungarian algorithm (transfer_labels.py in the dataset repository). That transfer agrees with the human-accepted corrections of the matched release to a mean centre error of 4.31 px, with 95.9% of boxes at IoU > 0.5.

Frames were filtered by the matching head of bambi-occlusion-classifiers -- non_geo/occlusion_matched.pt for the perspective arm (keeps 55%), and the geo_1k and geo_2k heads for the orthographic ones (52% and 54%). Each set of released weights is retrained on all of its flights; the tables above are the cross-validated estimate of their accuracy, not a measurement of these particular weights.

Orthographic renders were produced with alfs_pytorch at a 70 m window, 1024 px (geo_1k) and 2048 px (geo_2k), projected onto the per-flight terrain DEM. Three flights are absent from the orthographic arm: two whose DEM files are corrupt, and one whose box corners ray-cast onto the DEM tens of metres apart instead of two or three.

Limitations

• Feature-space locked to DINOv3 ViT-H+. A different backbone, or a different crop convention, will silently degrade accuracy.
• Projection-specific. The five projections are not interchangeable; each head expects the feature distribution of its own render.
• The ALFS heads are the weakest per frame and are not a general-purpose choice. They are for canopy-heavy material where the animal is intermittently hidden, and they want unfiltered input.
• Per frame the perspective heads are the strongest; per track, geo_2k is. Prefer non_geo for frame-level decisions on native imagery, geo_2k for pipelines already working in projected space. geo_1k is kept for continuity with the occlusion release but is superseded by geo_2k at the same inference cost.
• The matched head needs genuinely co-registered pairs. Feeding it an RGB box that was not transferred onto the animal will hurt more than using the RGB head alone.
• Three species only. Anything else -- fallow deer, chamois, ibex, birds, humans -- will be forced into one of the three. Fallow deer is the largest excluded class in BAMBI (23,636 boxes) and is a plausible source of confident errors.
• Roe deer is the weakest class at 87%, and the smallest in the training data (5,218 boxes against 17,038 and 13,835).
• Balanced accuracy near 0.9 suits triage and pre-labelling, not unreviewed population statistics.

Licence

The heads are MIT. The DINOv3 backbone carries Meta's separate licence terms; verify compliance independently.

Provenance

The species models built for the paper below were unfortunately lost, so these were retrained from scratch on the same data with the same recipe -- frozen DINOv3 ViT-H+ features, a batch-hard triplet projection to 256-d, then a linear classifier.

The retraining was checked against the one component that did survive. Rebuilt with this code, the occlusion heads reproduce the released bambi-occlusion-classifiers closely across all three projections:

Within about a point in each case, with the same ordering across modalities, so the pipeline behind these species heads is faithful to the original.

Citation

@misc{markoff2026multimodal,
  title  = {When One Modality Is Not Enough: Multimodal Sex and Life-Stage
            Classification of Red Deer from Aerial RGB-Thermal Video},
  author = {Markoff, Hugo and Praschl, Christoph and Ludo{\v{s}}ki, Ivan and
            Beery, Sara and {\O}rsted, Michael and Schedl, David C.},
  year   = {2026},
  eprint = {2608.06973},
  archivePrefix = {arXiv}
}