fusion-perception-1-preview

fusion-perception-1 (v0.1-preview)

<p align="center"> <img src="assets/fusion-perception-banner.png" alt="Fusion Perception — place recognition, Eximius Labs" width="100%"> </p>

A landmark and place retrieval descriptor: a frozen DINOv2 ViT-L/14 backbone (Apache-2.0) with a small trained projection head (3.1M parameters, the only weights in this repository). One image in, one 512-d unit vector out. Rank a gallery by cosine similarity.

Five queries against the full evaluation gallery, single-stage cosine search with no reranking. The first four are the shipped head working; the last is a query it gets wrong.

The intended use is re-localization and place matching at scale, the case a semantic search embedding handles poorly: same building, different viewpoint, different season, different time of day, among a million unrelated photographs.

Headline. With AMES reranking of a top-1600 shortlist, this system sets the best RParis+1M numbers we are aware of, on Medium and on Hard, under both of its training protocols. On ROxford+1M it is close to the frontier but not at it. Both statements are laid out below, and so is the reason to discount part of the result.

Why not a semantic search embedding

A multimodal search embedding already produces one vector per image, so the first question is whether a separate descriptor is needed at all. We measured it. Our own text, image, video and audio model, fusion-embedding-2 (revision 1720d8b1), was scored on both benchmarks with the same evaluator, the same official ground-truth files, the same bounding-box-cropped queries and the same galleries. Its image path is the frozen Qwen3-VL-Embedding-2B vision-language model, called through that model's released embed_image API. Global descriptors only, no reranking, no distractors.

The search embedding is behind on all four cells: 28.99 mAP-M and 39.40 mAP-H on ROxford, 14.43 and 27.82 on RParis, against the head in this repository. It is also behind the untrained DINOv2 readout by 25.42 / 35.06 and 14.16 / 26.96, so the difference is not something the head training created. The gap is widest on Hard, the setting whose positives are the difficult views of a landmark, taken from another viewpoint or under occlusion or different conditions, which is the case re-localization actually runs into.

This is a statement about one task. The search embedding is built for cross-modal retrieval and is not trained for instance-level matching, and nothing here measures the tasks it is built for.

Read-out note: twelve documented configurations of the search embedding were measured, crossing the document instruction on both sides against the query instruction on the query side, the 1024-d default against the full 2048-d rung, and gallery mean-centering on against off. The row above is its strongest configuration with protocol-cropped queries. Its weakest cropped configuration scores 38.20 M on ROxford and 70.18 M on RParis. The only cell where any configuration beats the row above is ROxford Hard under an uncropped-query control, which is outside the protocol and reaches 22.71, still 35.38 below the head. All twelve are recorded in results.json.

Results with +1M distractors

mAP in percent under the revisited Oxford / Paris protocol, with the standard 1,001,001-image revisitop1m distractor set. Two-stage rows all rerank a top-1600 shortlist; shortlist size is the single largest confound in this table, so rows reranking 400 or 800 candidates are left out rather than compared against.

Our rows are one training run each. The seed spread measured for this recipe is given under Evaluation integrity; on RParis the shipped standard head is the weakest of four runs, so these rows are not a favourable draw.

RParis+1M. Both heads pass the best published numbers on both difficulty settings, by +0.51 M and +2.63 H for the standard head and +0.82 M and +2.95 H for the decontaminated head, measured against L2G. The two heads are not separable from each other at this sample size, so the result to take from the pair is that it holds whether or not the overlapping training classes are removed.

ROxford+1M. Medium is level with AMES on DINOv2 locals without SuperGlobal reranking (87.21 and 87.11 against 87.1) and 3.3 to 3.4 points below L2G. Hard is 1.0 to 2.4 points below that same AMES row and 4.7 to 6.1 points below L2G. ROxford is not a result we lead on.

What the reranker contributes. The global-only rows are in the table so the split is visible. On ROxford+1M, reranking a top-1600 shortlist adds 20.1 mAP-M over our own first-stage ranking; on RParis+1M it adds 6.0. The asymmetry has a first-stage cause: our global descriptor scores 67.09 M on ROxford+1M against RN101-SuperGlobal's 78.8, while on RParis+1M it scores 86.65 against their 83.9. The reranker starts from a much worse shortlist on ROxford and from a better one on RParis.

Sources: [1] SuperGlobal, ICCV 2023, arXiv 2308.06954, Table 1 and supplementary Table 6. [2] AMES, ECCV 2024, arXiv 2408.03282, supplementary Table 9. [3] L2G, ICCV 2025 Workshop (MRR), arXiv 2509.04351, Table 1.

One citation subtlety worth stating: the AMES paper's main Table 1 reports only the collapsed ROP+1M average for these configurations. The four per-dataset +1M cells quoted above are in supplementary Table 9, not the main table. L2G's Table 1 reproduces the AMES best row and agrees with it.

Evaluation integrity

Two contamination channels affect these numbers. Both are stated here rather than in an appendix, because a reader deciding whether to trust the table needs both.

1. Training-set overlap with the benchmarks

We matched GLDv2-clean category names against the ROxford and RParis query landmarks and excluded 87 classes containing 2,529 images from the decontaminated run. We then checked that match set at image level rather than asserting it, and it does not hold up as a contamination measurement.

Embedding every flagged image and all 140 bounding-box-cropped benchmark queries with the frozen DINOv2-L descriptor, and calibrating the decision threshold on the benchmarks' own ground truth (1% false-positive rate against database images of other landmarks, 69% true-positive rate against known same-landmark positives), 43% of the flagged images actually depict a landmark the benchmarks query. A visual audit of 170 randomly sampled flagged images, judged against the query crops, gives the same answer: 43% plus or minus 8%, agreeing with the automatic verdict at Cohen's kappa 0.84. At class level, 13 of the 87 classes are a queried landmark, 23 if structures inside the same complex are counted.

Precision varies sharply by rule. The unique:<landmark> rules reach 58% at image level. The blanket:oxford rule, which covers half the set, reaches 18%: it excludes Oxford Street and Oxford Circus in London, the Selfridges Oxford Street store, an Oxford College in Taiwan and villages across Oxfordshire. The ctx:paris+notre-dame rules reach 8%: thirteen of their fifteen classes are Paris parish churches named Notre-Dame that the benchmarks never query.

The set is also not a superset of the published cleanup. RGLDv2-clean (Song et al., CVPR 2024) removes 1,565 images from 18 landmark ids; our 87 classes contain 10 of those 18 and 1,216 of those 1,565 images. The eight we miss include Basilique du Sacre-Coeur de Montmartre, whose Commons category spells Coeur with the oe ligature and so slipped past a rule that flagged five other Sacre-Coeur basilicas. Running the same calibrated detector over all of GLDv2-clean finds 995 images at a queried landmark; the 87 classes hold 757 of them, a recall of 76%.

The decon head is therefore a conservative ablation, not a cleaner benchmark protocol: it removes the overlapping classes it does catch plus roughly 1,400 images unrelated to the benchmarks, so the decon-versus-standard gap bounds the cost of removing overlapping training data from above. All 87 classes and their matching rules remain listed in leak_class_ids.json. The detectors behind every number above, their calibrations and the per-image scores are in audit/: rgld_detector.json for the DINOv2 detector, clip_detector.json for the CLIP replication, leak_raw_scores.json for the per-image similarities against all 140 query crops, and rgld_removal_set_diff.json for the RGLDv2-clean removal set reconstructed from the authors' released CSV.

We therefore ship two heads:

• standard trains on all of GLDv2-clean (1,570,970 images, 76,563 classes after the minimum-three-images-per-class filter). This is the same training data every published system in the table above uses, so it is the row to compare against them.
• decon removes those 87 classes first (1,568,445 images, 76,478 classes).

The measured effect is smaller than the run-to-run variation of the training itself, so we do not claim a direction for it. Retraining both heads with three seeds each, varying nothing but the seed, gives these no-distractor global-only means with sample standard deviations over the three runs:

Three of the four differences are smaller than the seed spread within a condition, and the fourth is 1.1 pooled standard deviations, which an exact permutation test at three runs per condition cannot separate from zero. Two of them reverse sign relative to the single-run numbers the shipped heads produced. Across the six runs ROxford Medium and RParis Medium are negatively correlated (r = -0.74), which is expected because the epoch is selected on ROxford, so the apparent trade between the two datasets in a single-run comparison is largely that selection effect.

What this supports is a bound rather than a null result: removing the 87 overlapping classes does not change these cells by more than roughly 1 mAP in either direction. It does not explain the RParis result, which sits 16 points above the ROxford figure.

The shipped heads are one run each and were trained without a fixed seed, so the spread above is the error bar to read every single-run number in this card with. On RParis the shipped standard head is the weakest of the four runs measured (92.85 M against 93.47, 93.15 and 92.97; 84.88 H against 86.27, 85.56 and 85.29), so the reranked RParis rows this release leads with come from an unfavourable draw rather than a fortunate one. On ROxford Hard the shipped head is the strongest of the four (58.09 against 55.47, 56.78 and 57.78) and should be read as such. Per-seed results are in audit/seed_analysis_n6.json, which carries both the three-seed statistics quoted here and a six-seed extension of them.

2. The backbone saw the benchmarks during pretraining

DINOv2's pretraining corpus LVD-142M contains GLDv2-clean as is (1,580,470 images) plus 6,321,880 images sampled from it, and it contains 1,000,000 images retrieved using the Revisiting Oxford base set as seeds and another 1,000,000 retrieved using the Revisiting Paris base set (DINOv2, arXiv 2304.07193, Table 15). The backbone is frozen here, so this is not something our training introduced, but it is exposure our descriptor benefits from and it cannot be removed without changing the backbone.

The AMES authors declined DINOv2 as their default global descriptor for exactly this reason:

We do not consider DINOv2 as the default option since ROxford and RParis are listed among the datasets used for its self-supervised training.

— AMES, arXiv 2408.03282, section 4

This creates an asymmetry in the table above, and it runs against us. Our system uses DINOv2 for the global descriptor and DINOv2 local descriptors for reranking, so the caveat applies to it in full. It also applies to the two AMES rows built on DINOv2 locals, including their strongest +1M row. It does not apply to L2G, which uses FIRE local features and no DINOv2, nor to SuperGlobal, nor to the AMES row built on CVNet locals. The strongest published row we exceed on RParis+1M is therefore a row that carries less pretraining exposure than ours does. A reader who discounts DINOv2-based results should discount ours first.

Results without distractors

Same protocol, database only, for readers who want the comparison at the original benchmark scale.

The pattern is the same and less pronounced: RParis competitive with the best published rows, ROxford several points behind them. The margin on RParis widens once the 1M distractors are added, which is the useful direction for a system meant to run against a large index.

Architecture

  image
    ├─ scale 1.000  short side 224 px ─┐
    ├─ scale 1.414  short side 322 px ─┤  frozen DINOv2 ViT-L/14 (Apache-2.0)
    └─ scale 2.000  short side 448 px ─┘  CLS token, L2-normalized per scale
                                       │
                          mean of 3, re-normalized -> 1024-d
                                       │
       Linear(1024, 2048) -> GELU -> Linear(2048, 512) -> BatchNorm -> L2
                                       │
                              512-d retrieval descriptor

Aspect ratio is preserved; each side is snapped to a multiple of 14 and the long side is capped at 1022 px. The head is trained with ArcFace (margin 0.3, scale 32), label smoothing 0.1, class-balanced square-root sampling, AdamW with a cosine schedule, batch 4096, 40 epochs with early stopping. Model selection is on ROxford only; RParis was held out until the final table. The backbone is never fine-tuned.

Reranking is optional. The two-stage rows use AMES (Suma et al., ECCV 2024) with the authors' public checkpoint and their published DINOv2-B local descriptors, unchanged, over a top-1600 shortlist, combined as 0.55 * global_cosine + 0.45 * sigmoid(0.3 * ames_logit). That is the AMES paper default. Our contribution to those rows is the first-stage ranking.

rerank.py in this repository runs that second stage, so the two-stage cells above are reproducible from what ships here. The AMES code is Apache-2.0 and is fetched with torch.hub; their checkpoint and their local descriptors download from the authors' host on first use. None of it is redistributed here, and the local descriptors are DINOv2-B features, so the pretraining caveat above applies to the two-stage rows as well.

Usage

<details> <summary>Requirements</summary>

torch, transformers>=4.46, safetensors, pillow, numpy, and huggingface_hub if loading by repo id. The DINOv2 backbone downloads from Meta's repository under Apache-2.0. Runs on CPU; float16 on CUDA is the precision every reported number was produced at.

</details>

from PIL import Image
from inference import FusionPerceptionRetrieval

fp = FusionPerceptionRetrieval.from_pretrained("EximiusLabs/fusion-perception-1-preview")

gallery = fp.embed([Image.open(p) for p in paths])   # [N, 512], L2-normalized
query   = fp.embed(Image.open("query.jpg"))          # [512]

scores, idx = fp.search(query, gallery, topk=10)

Select the decontaminated head with protocol="decon". Pass bbox=[(x1, y1, x2, y2), ...] to crop before embedding, which is what the revisitop query protocol does.

Which head to use: standard if you are comparing against published numbers, decon if your application is anywhere near Oxford or Paris landmarks and you want the training set to have no knowledge of them. For everything else the two are close enough that the choice does not matter much.

Two stage

# once per dataset: pull the AMES authors' local descriptors, 5.5 GB here, 7 GB for RParis
python rerank.py --dataset roxford5k --ames-dir ./ames_assets --fetch-only

# rerank the top-1600 shortlist and score Medium and Hard
python rerank.py --dataset roxford5k --ames-dir ./ames_assets \
    --images-root /data/roxford5k/jpg --head standard --out roxford5k_standard.json

Defaults are the reported cell: --topk 1600 --lambdas 0.55 --temps 0.3. Pass --descriptors instead of --images-root to reuse descriptors you already have, and --distractor-locals with --distractor-desc for the +1M setting. The script needs h5py and a GPU in addition to the requirements above.

Reproduction

REPRODUCE.md in this repository lists the scripts, the evaluation harness, the exact two-stage commands, where the distractor features come from, and the compute and storage the full run required (roughly $100 of GPU time and 1.08 TB of local-descriptor storage). The no-distractor two-stage rows in the table above were reproduced with rerank.py and match to the last reported digit; the figures and the one deviation worth knowing about are in REPRODUCE.md.

Scope and limitations

• This release ships the retrieval descriptor only. Dense segmentation heads and a projector into the Fusion Embedding shared space exist in the same line of work and are not included here.
• Research preview. Evaluated on ROxford and RParis, which are landmark benchmarks: performance on indoor scenes, repetitive industrial environments, or low-texture surfaces is not characterized by these numbers.
• The strong result is specific to RParis+1M. This is not a general state-of-the-art claim.
• Reranking is what closes most of the gap to published two-stage systems, and it costs memory for stored local descriptors. The global-only rows are the operating point with no extra storage cost, and they are several points weaker.
• The open backbone is DINOv2 (Apache-2.0). A DINOv3-based variant is stronger on some tasks but is licensed in a way that does not permit an open-weights release.

License

Released for research under CC-BY-NC-4.0. The DINOv2 backbone is Apache-2.0 and is downloaded from Meta rather than re-hosted here. The AMES code (pavelsuma/ames) is Apache-2.0 and is fetched from its repository; the AMES checkpoint and local descriptors used for the two-stage rows belong to their authors, are downloaded from the authors' host, and are not redistributed. rerank.py calls their code rather than reimplementing it. Cite AMES for any use of the two-stage numbers.

Citation

@misc{fusionperception2026,
  title  = {Fusion Perception: a frozen-backbone retrieval descriptor for place recognition},
  author = {Tonmoy, Abdul Basit},
  year   = {2026},
  note   = {Eximius Labs. DINOv2 backbone (Apache-2.0); head CC-BY-NC-4.0.},
  url    = {https://huggingface.co/EximiusLabs/fusion-perception-1-preview}
}

Backbone: DINOv2 (Oquab et al., 2023). Benchmarks: Revisited Oxford and Paris (Radenović et al., CVPR 2018) with the revisitop1m distractor set. Reranking: AMES (Suma et al., ECCV 2024).