satellite-omni-lisa

🛰️ Satellite Omni LISA

A multi-modal satellite model that understands AND generates images.

Architecture

Based on LISA: Reasoning Segmentation via Large Language Model, adapted for satellite/remote sensing with multi-task training. Uses SlimSAM (9.7M params, 77% pruned) instead of SAM-ViT-B (93.7M) for efficiency.

┌─────────────────────────────────────────────────────────┐
│                    INPUT                                 │
│  Satellite Image + Text Prompt                          │
│  e.g., "Segment the buildings in this aerial image"     │
└────────────────────┬────────────────────────────────────┘
                     │
┌────────────────────▼────────────────────────────────────┐
│             Qwen3.5-0.8B VLM (LoRA fine-tuned)          │
│  ┌────────────┐  ┌───────────────────────────────────┐  │
│  │ ViT Vision │  │ Language Model (24 layers, 1024d)  │  │
│  │  Encoder   │──│ Linear + Full Attention layers     │  │
│  │  (768d)    │  │ MRoPE positional encoding          │  │
│  └────────────┘  └──────────┬────────────────────────┘  │
└─────────────────────────────┼───────────────────────────┘
                              │
              ┌───────────────┴──────────────┐
              │                              │
     ┌────────▼─────────┐          ┌─────────▼──────────┐
     │   Text Output     │          │  <SEG> Token        │
     │                   │          │  Hidden State       │
     │  "I identified    │          │  (1024-dim)         │
     │   5 buildings"    │          └─────────┬──────────┘
     └───────────────────┘                    │
                                    ┌─────────▼──────────┐
                                    │  MLP Projection     │
                                    │  1024 → 256         │
                                    │  (GELU activation)  │
                                    └─────────┬──────────┘
                                              │
     ┌────────────────────────────────────────▼──────────┐
     │            SlimSAM-77                              │
     │  ┌─────────────┐  ┌────────────────────────────┐  │
     │  │ ViT Encoder  │  │ Mask Decoder (fine-tuned)  │  │
     │  │ (frozen)     │──│ Cross-attention + MLP      │  │
     │  │ ~5.7M params │  │ ~4M params                 │  │
     │  └─────────────┘  └──────────┬─────────────────┘  │
     └───────────────────────────────┼───────────────────┘
                                     │
                           ┌─────────▼──────────┐
                           │  Binary Mask Image  │
                           │  (256×256 → resize) │
                           └─────────────────────┘

Key Innovation

The model handles two output modalities from a single VLM:

1. Text output: Standard autoregressive text generation for captioning, VQA, classification
2. Image output: When the model generates a special <SEG> token, its hidden representation is projected through an MLP and used as a prompt for SlimSAM's mask decoder to generate a segmentation mask image

This means the model learns when to generate an image (by outputting <SEG>) and what image to generate (the hidden state encodes the full visual context).

Tasks

Text Output Tasks

Image Output Tasks

Training Details

Loss Function

L = L_text + L_mask
L_text = CrossEntropy(predicted_tokens, target_tokens)
L_mask = 2.0 * BCE(predicted_mask, gt_mask) + 0.5 * DICE(predicted_mask, gt_mask)

Hyperparameters

Parameter Budget

Dataset

Trained on rahuldshetty/satellite-multitask-omni:

• ~20K training samples across 9 task types
• Sources: EuroSAT, RSICD, PatternNet, FloodNet, building segmentation datasets
• Segmentation masks from normalized polygon coordinates (0-1000 range)

Usage

Installation

# Qwen3.5 requires transformers from git (>= 4.57.0)
pip install "transformers @ git+https://github.com/huggingface/transformers.git"
pip install torch torchvision peft datasets Pillow numpy huggingface_hub

# Or use the setup script:
bash setup.sh

Single-Image Inference

from inference import load_model, generate_text, generate_segmentation
from PIL import Image

# Load model
model = load_model("./model_dir", device="cuda")

# Text task
image = Image.open("satellite.jpg")
response = generate_text(model, image, "Describe this satellite image")
print(response)

# Segmentation task
text, mask = generate_segmentation(model, image, "Segment the buildings")
print(text)
mask.save("building_mask.png")

CLI Inference

# Text generation
python inference.py --model_dir ./model --image sat.jpg --prompt "What land cover type?"

# Segmentation (generates mask image)
python inference.py --model_dir ./model --image sat.jpg --prompt "Segment buildings" --output_mask mask.png

Dataset Evaluation

Run inference on sampled dataset examples across all task types and generate a structured report:

# Default: 3 samples per task from validation split
python infer_dataset.py --model_repo rahuldshetty/satellite-omni-lisa

# More samples, custom output
python infer_dataset.py --model_repo rahuldshetty/satellite-omni-lisa \
    --samples_per_task 5 \
    --split test \
    --output_dir ./my_eval

# Force CPU (slower but works without GPU)
python infer_dataset.py --device cpu --samples_per_task 2

This produces:

eval_output/
├── REPORT.md              ← Markdown report with all results
├── results.json           ← Machine-readable results
└── images/
    ├── sample_000_input.png
    ├── sample_003_gt_mask.png
    ├── sample_003_pred_mask.png
    └── ...

The REPORT.md includes:

• Summary table — per-task sample counts, error counts, mean IoU for segmentation
• Per-task sections — each sample shows:
  • Input satellite image
  • Text tasks: expected vs model output in a comparison table
  • Segmentation tasks: GT mask vs predicted mask side-by-side with IoU score

Training

# Auto-detects GPU, sets batch size, bf16/fp16 automatically
python train.py

# Override settings via environment variables
BATCH_SIZE=1 GRAD_ACCUM=16 MAX_SAMPLES=100 python train.py

Files

References

• LISA: Reasoning Segmentation via Large Language Model (CVPR 2024)
• SlimSAM: 0.1% Data Makes Segment Anything Slim
• LISAT: Remote Sensing Adaptation
• Qwen3.5-0.8B
• SAM: Segment Anything