ExPLoRA

ExPLoRA: Parameter-Efficient Extended Pre-Training

Paper | Code | Website | Video

This repository contains pre-trained checkpoints from the ICML 2025 paper:
"ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts"

Overview

ExPLoRA is a parameter-efficient method for adapting pre-trained Vision Transformers (ViT) to new domains using LoRA-based extended pre-training. Instead of training the full architecture, ExPLoRA freezes most of the backbone and trains low-rank adapters and a small subset of ViT blocks during self-supervised pre-training on target domain data.

<p align="center"> <img src="https://samar-khanna.github.io/ExPLoRA/static/images/explora_arch.svg" width="600" style="background-color: white; padding: 10px; border-radius: 8px;"/> </p>

---

📁 Checkpoints

Note: All checkpoints have LoRA adapters already merged into the weights. The full checkpoints retain the separate q_proj, k_proj, v_proj layers (with merged LoRA) alongside the combined qkv weights for reference. The encoder-only checkpoints contain just the merged qkv weights, ready for downstream use.

explora_dinov2_fmow_rgb/

ExPLoRA checkpoints using DINOv2 self-supervised pre-training on fMoW high-resolution RGB satellite imagery.

Usage:

import torch

# Load encoder-only checkpoint (recommended for fine-tuning)
ckpt = torch.load("explora_dinov2_fmow_rgb/explora_dinov2_vit_large_fmow_rgb_encoder_only.pth", map_location="cpu")
state_dict = ckpt["model"]

explora_mae_multispectral/

ExPLoRA checkpoints using MAE self-supervised pre-training on fMoW Sentinel-2 multispectral imagery.

Usage:

import torch

# Load encoder-only checkpoint (recommended for fine-tuning)
ckpt = torch.load("explora_mae_multispectral/explora_mae_fmow_sentinel_encoder_only.pth", map_location="cpu")
state_dict = ckpt["model"]

---

Loading Checkpoints

These checkpoints are compatible with the ExPLoRA codebase.

For fine-tuning, use the finetune/finetune.py script:

python finetune/finetune.py \
    --finetune path/to/explora_checkpoint.pth \
    --model vit_large_patch16 \
    --dataset_type rgb \
    ...

Reference scripts are also provided under scripts/ in the codebase, and you can use these checkpoints there.

---

Citation

If you find these checkpoints useful, please cite our paper:

@inproceedings{khanna2025explora,
  title={Ex{PL}o{RA}: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts},
  author={Samar Khanna and Medhanie Irgau and David B. Lobell and Stefano Ermon},
  booktitle={Forty-second International Conference on Machine Learning},
  year={2025},
  url={https://openreview.net/forum?id=OtxLhobhwb}
}

License

Apache 2.0