citysense-gpt

CitySenseGPT

CitySenseGPT: A Vision-Language Model for Semantic Reasoning of the Built Environment via Satellite Imagery

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama community license for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

Contents

• Install
• Table/Figure
• Training

Install

conda create -n citysense-gpt python=3.10.14 -y
conda activate citysense-gpt
pip install "numpy<2"
pip install torch==2.1.2+cu121 torchvision==0.16.2+cu121 torchaudio==2.1.2+cu121   --extra-index-url https://download.pytorch.org/whl/cu121
cd inference
pip install -r requirements.txt
cd ..
pip install -e .
pip install "protobuf>=3.20.0,<5"

Table and Figure Generation

The folder of table_01 contains all scripts to generate table 1 in the paper.

Quantitative Reasnoning Test

cd table_01
cd quantitative_reasoning
python grading_citysense.py
python grading_llava.py
python grading_internvl.py
python grading_geochat.py
python grading_vhm.py

Qualitative Reasnoning Test

pip install nltk
pip install rouge_score
pip install pycocoevalcap
pip install rapidfuzz
pip install openpyxl
cd table_01
cd qualitative_reasoning
python grading_citysense_text.py
python grading_llava_text.py
python grading_internvl_text.py
python grading_geochat_text.py
python grading_vhm_text.py

The folder of Figures contains all scripts to generate figures 5-9 in the paper.

With respect to figures 5 and 6, the generated figures with .png are saved in the same directory.(Note: Back to relative home directory!)

cd Figures/Figures_56.
python figures_5_6.py

With respect to figures 7, 8, and 9. The generated results are stored in vqa_inference_results_789.jsonl. The answers appear in the text field. This file is a text-based format and can be opened with Notepad. (Note: Back to relative home directory! )

pip install absl-py
cd Figures
sh citysense-gpt-inference.sh

Training

We provide the training scripts, including citysense-gpt-finetune.sh and citysense-gpt-lora-merge.sh. It should be noted that the training configurations are compatible with B200 GPUs.

sh citysense-gpt-finetune.sh
sh citysense-gpt-lora-merge.sh

Training data set is shared through the link : https://drive.google.com/file/d/1D8zjGqoFON1zrgeKhsQDvdVcBXgmAPS6/view?usp=sharing

Evaluation

Go to the folder of inference. NOTE: The code was tested on L4 GPUs. Using higher-end GPUs may require additional configuration and could lead to compatibility issues.

python ../model_vqa.py \
    --num-chunks 1 \
    --model-path ../checkpoints/llava-v1.5-7b-lora-citysense-gpt-ft-merged \
    --question-file \
    ./citysenseGPT_QA_data_llava_test-limit-words.jsonl  \
    --image-folder \
    ../rs-images-test  \
    --answers-file \
    ./vqa_inference_results_citysense-gpt.jsonl

⚠️ Important: Fix for Windows Download Errors (OSError 1314)

If you encounter OSError: [WinError 1314] when downloading the 14.1GB dataset or model checkpoints, it is a Windows-specific permission issue related to symbolic links.

To resolve this, please use one of the following methods:

1. Enable Developer Mode (Recommended): - Open Windows Settings > Privacy & Security (or Update & Security) > For developers.
   • Toggle Developer Mode to On.
2. Run as Administrator: - Right-click your terminal (PowerShell or CMD) or IDE and select "Run as Administrator".
3. Environment Variable Bypass:
   • Run the command set HF_HUB_DISABLE_SYMLINKS=1 in your terminal before execution, or add import os; os.environ["HF_HUB_DISABLE_SYMLINKS"] = "1" at the very top of your Python script.