osmclip_vit32base

🌍 OSM-CLIP / OpenStreetCLIP (ViT-B/32)

Official weights for OSM-CLIP (OpenStreetCLIP), a vision–language framework that leverages OpenStreetMap (OSM) annotations for patch-level geographic supervision in remote sensing.

🚀 Overview

Standard CLIP models operate on global image–text alignment, collapsing an entire satellite image into a single feature vector. OSM-CLIP instead injects patch-level geographic semantics by aligning ViT patches with OSM-derived textual tags (e.g. building residential, highway primary), enabling spatially grounded representations that better capture fine-grained geospatial structure.

This repository provides all three checkpoints used in the ablation study of the associated paper.

📦 Available Checkpoints

All three models share the same architecture and fine-tuning setup; only the pre-training strategy changes.

🛠 Technical Specifications

• Backbone: ViT-B/32 (CLIP image encoder)
• Text encoder: CLIP text transformer
• Input size: 224 × 224 (patch grid 7 × 7 for ViT-B/32)
• Framework: PyTorch
• Pre-training (OSM-PT):
  • Patch-level contrastive loss between aggregated patch embeddings and OSM tag embeddings
  • ~265k images over the contiguous US with OSM-derived annotations
• Fine-tuning (FT):
  • Standard CLIP contrastive loss on RS captioning datasets (NWPU-RESISC45, RSICD, UCM, SYDNEY)
  • Optimizer: Adafactor
  • Single run per configuration (no multi-run ensembling)

🧪 Recommended Usage

• Use osm-clip-pretrained-finetuned.pt when you want spatially grounded, OSM-aware representations for:

  • zero-shot or linear-probe RS scene classification,
  • image–text retrieval,
  • geospatial grounding / visualization of patch activations.

• Use clip-standard-finetuned.pt as a strong CLIP-based baseline without any RS- or OSM-specific pre-training.

• Use clip-rs-pretrained-finetuned.pt if you want to replicate the ablation from the paper and test the effect of “more RS data without OSM semantics”.

📜 Licenses & Attribution

• Code & Weights: Licensed under GPL-3.0.
• OSM Data: This model was trained using data from OpenStreetMap.
  OpenStreetMap data is licensed under the Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF).
  © OpenStreetMap contributors.

📖 Citation

If you use these models, please cite the associated paper (currently under review):

@article{openstreetclip2026,
  title   = {OSM-CLIP: Enhancing Remote Sensing Image-Text Representation Learning with OpenStreetMap Data},
  author  = {Pierdominici, Alessio and Ricci, Riccardo and Alruqimi, Mohammed and Melgani, Farid},
  journal = {Applied Sciences},
  year    = {2026},
  note    = {submitted}
}

(Update with volume/issue/DOI once published.)

🔗 Links

• Code (training & evaluation): https://github.com/alessiopierdominici/OpenStreetCLIP
• Dataset (OSM-CLIP pre-training): https://huggingface.co/datasets/alessiopierdominici/osm-clip-dataset

Developed at the University of Trento.