satvision-base

SatVision-Base A vision transformer trained with MODIS surface reflectance data

• Developed by: NASA GSFC CISTO Data Science Group
• Model type: Pre-trained vision transformer model
• License: Apache license 2.0

SatelliteVision-Base (SatVis-B) is a pre-trained vision transformer based on the SwinV2 model architecture. The model is pre-trained on global MODIS surface reflectance data from which 1.99 million image chips were used. SatVis-B is pre-trained using the masked-image-modeling (MIM) contrastive pre-training strategy. The MIM pre-training approach utilizes random masking of the input geospatial image chip, using a linear layer to regress the raw pixel values of the masked area with an l1 loss serving as the loss function.

Resolution of the pre-training MODIS chips was 128x128 with a window size of 16x16. SatViz-B was pre-trained for 800 epochs on 8x A100 GPUs and 12x V100 GPUs.

SatVision Transformer

Pre-trained models pre-trained on MODIS-Small dataset

Getting Started with SatVision-Base

• Training repository: https://github.com/nasa-nccs-hpda/pytorch-caney
• Pre-training dataset repository: https://huggingface.co/datasets/nasa-cisto-data-science-group/satvision-pretrain-small

Installation

If you have singularity installed

$ git clone [email protected]:nasa-nccs-hpda/pytorch-caney.git
$ singularity build --sandbox pytorch-caney.sif docker://nasanccs/pytorch-caney:latest
# To shell into the container
$ singularity shell --nv -B <mounts> pytorch-caney.sif

Anaconda installation

$ git clone [email protected]:nasa-nccs-hpda/pytorch-caney.git
$ conda create -n satvision-env python==3.9

Fine-tuning Satvision-Base

• Create config file example config
• Download checkpoint from this HF model repo
• $ git clone [email protected]:nasa-nccs-hpda/pytorch-caney.git
• Add a new pytorch dataset in pytorch-caney/pytorch_caney/data/datasets/
• Add new pytorch dataset to dict in pytorch-caney/pytorch_caney/data/datamodules/finetune_datamodule.py

torchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/finetuning/finetune.py --cfg <config-file> --pretrained <path-to-pretrained> --dataset <dataset-name (key for new dataset)> --data-paths <path-to-data-dir> --batch-size <batch-size> --output <output-dir> --enable-amp

Pre-training with pytorch-caney

Pre-training with SatVision-Base with Masked Image Modeling and pytorch-caney

To pre-train the swinv2 base model with masked image modeling pre-training, run:

torchrun --nproc_per_node <NGPUS> pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg <config-file> --dataset <dataset-name> --data-paths <path-to-data-subfolder-1> --batch-size <batch-size> --output <output-dir> --enable-amp

For example to run on a compute node with 4 GPUs and a batch size of 128 on the MODIS SatVision pre-training dataset with a base swinv2 model, run:

singularity shell --nv -B <mounts> /path/to/container/pytorch-caney-container
Singularity> export PYTHONPATH=$PWD:$PWD/pytorch-caney
Singularity> torchrun --nproc_per_node 4 pytorch-caney/pytorch_caney/pipelines/pretraining/mim.py --cfg pytorch-caney/examples/satvision/mim_pretrain_swinv2_satvision_base_192_window12_800ep.yaml --dataset MODIS --data-paths /explore/nobackup/projects/ilab/data/satvision/pretraining/training_* --batch-size 128 --output . --enable-amp

SatVision-Base Pre-Training Datasets

Citing SatVision-Base

If this model helped your research, please cite satvision-base in your publications.

@misc{satvision-base,
	author       = {Carroll, Mark and Li, Jian and Spradlin, Caleb and Caraballo-Vega, Jordan},
	doi          = {10.57967/hf/1017},
    month        = aug,
	title        = {{satvision-base}},
	url          = {https://huggingface.co/nasa-cisto-data-science-group/satvision-base},
    repository-code = {https://github.com/nasa-nccs-hpda/pytorch-caney}
    year         = {2023}
}