TSRDA

T-SRDA — Temporal Spatial-Reduction Dual-Attention for Crop Mapping

T-SRDA swaps the Swin/DaViT windowed temporal encoder of an AD-STCLN-style crop-mapping pipeline for a Spatial-Reduction Dual-Attention (SRDA) encoder — adapted from PDAViT (Zhou et al., Neurocomputing 2026) — applied along the temporal axis instead of the spatial axis it was designed for. The idea: reduce a satellite image time series to a small set of "key-value" summary tokens per pixel, then let every timestep attend to that reduced set in a single global cross-attention layer, instead of needing a stack of shifted local windows to approximate a global receptive field.

The pipeline is masked-reconstruction pretrained on unlabelled Sentinel-2 time series, then finetuned end-to-end for 18-class crop-type semantic segmentation on PASTIS, following the official STCLN protocol byte-for-byte (same seed, same folds, same steps/epoch, same hyperparameters) so the temporal encoder is the only intentionally-varied component.

Headline result: T-SRDA does not beat the baseline. Test mIoU 0.4124 vs. A_linear 0.4747 at the same seed (Δ = −0.0314, ≈4.4× the baseline's own seed-to-seed std of 0.0071). This is reported as a negative result — see § Results for the full picture, including three caveats that keep it from being conclusive (one seed, two residual architectural confounds, a checkpoint-selection mismatch).

This repository contains the full project: source code, training/eval scripts, every checkpoint from pretraining and finetuning, and the raw logs for every run.

---

Table of contents

1. Architecture
2. Dataset
3. Training configuration
4. Hardware & environment
5. Results
6. Training dynamics
7. Repository contents
8. Reproducing this run
9. Honest summary & next steps
10. Citation

---

Architecture

Input: (B, T, 10, H, W) Sentinel-2 time series + (B, T) acquisition-day offsets
        │
        ▼
┌───────────────────────────────────────────────────────────────┐
│  AtrousSpatialEncoder   (per-frame, shared across T)           │
│    dilated conv stem (d=64) → ASPP (dilations 1,2,4) → SE gate │
│    (B·T, 10, H, W) ──────────────────────────► (B·T, 256, H, W)│
└───────────────────────────────────────────────────────────────┘
        │  reshape to per-pixel sequences (B·H·W, T, 256)
        ▼
┌───────────────────────────────────────────────────────────────┐
│  TemporalSRDAEncoder   ← the architectural variable (T-SRDA)   │
│  3 × TemporalSRDABlock, R_schedule = [4, 4, 4]                 │
│                                                                  │
│    date PE added to x (sin/cos over real acquisition offsets)  │
│    for each block:                                              │
│      KV path : LayerNorm → Linear(d→d/R) → unfold R steps      │
│                → (T/R, d) → self-attention  ("self-checking")  │
│      Q  path : full-resolution T query tokens                  │
│      2nd attn: Q(T) × reduced-KV(T/R) cross-attention           │
│                → global temporal receptive field in ONE layer  │
│      scMLP   : FC → GELU + depthwise Conv1d(k=3, over T) → FC  │
└───────────────────────────────────────────────────────────────┘
        │  (B, H, W, T, 256)
        ▼
   ┌─────────────────────┐        ┌──────────────────────────────┐
   │  Pretrain: UTAEPrediction  │  │  Finetune: UTAEClassificationDual │
   │  NDVI-gated masked recon  │  │  STA temporal-attention pool      │
   │  Linear(256→10), MSE loss │  │  → semantic head (18+bg classes)  │
   │  4.01M params              │  │  → boundary head (morph. gradient)│
   │                             │  │  → sigmoid-gated residual refine  │
   │                             │  │  6.02M params                    │
   └─────────────────────┘        └──────────────────────────────┘

Why the temporal axis, not spatial? PDAViT's spatial-reduction dual-attention was designed to cut the quadratic cost of attention over image patches. Applied here to time instead of space, the same mechanism turns an irregular, variable-length acquisition sequence (33–61 Sentinel-2 revisits) into a small set of date-aware summary tokens, then gives every timestep a genuinely global view of the whole series in one layer — the Swin/DaViT baseline instead approximates this with a [4, 8, 16]-window + shift schedule.

Parameter count: temporal encoder alone 3.22M; full pretrain graph (spatial encoder + T-SRDA + reconstruction heads) 4.01M; full finetune graph (spatial encoder + T-SRDA + STA + dual decoder + gated refinement) 6.02M. For reference, the Swin variant of the same pipeline is ~2.71M and the DaViT variant ~3.50M (temporal encoder only).

What's held constant vs. the baseline (A_linear): everything except the temporal encoder was intended to be held constant, but two confounds remain in this arm — the decoder is dual (semantic + boundary + gated refinement) rather than a single linear head, and the spatial encoder is ASPP+SE rather than UTAE. Both are called out explicitly in § Results as caveats.

---

Dataset

PASTIS — Panoptic Agricultural Satellite TIme Series. Sentinel-2 optical time series over 2,433 patches (128×128 px, 10 spectral bands) in France, annotated with 18 crop types + background + void, at parcel-level panoptic granularity (only semantic segmentation is used here).

All four data roles are fold-disjoint (official 5-fold split, ~1 km spatial buffer between folds):

• Crop size for pretrain/finetune: 32×32 (PATCH_SIZE // 4); test runs on the full 128×128 patch with no cropping and no TTA.
• The model is fed torch.arange(T) index positions, not real day offsets (USE_INDEX_POSITIONS = True), matching the official reference implementation exactly — real acquisition days are still carried by the dataset loader for analysis.
• Train/val IDs are hardcoded lists, not a fold filter, ported verbatim from the official script — duplicates are intentional (some patches count twice per epoch).
• Pretext task: per-frame NDVI-gated masking (STCLN.py:193-202, ported bit-identical — verified max|Δ| = 0.0 across 3 seeds). NDVI = (NIR − Red)/(NIR + Red); a frame is exempted from masking entirely if ≤90% of its pixels are vegetated. Measured on 20 fold-5 patches: the gate fires on 90.6% of frames, leaving 96.2% of all values visible.

---

Training configuration

The fixed-epoch-99 checkpoint, not best-validation, is the reported primary result — see § Results for why.

T-SRDA-specific: KV reduction schedule R = [4, 4, 4] across 3 temporal blocks; 8 attention heads; d_model = 256; date positional encoding added before KV reduction (mandatory for irregular acquisition spacing); no internal gradient checkpointing in the temporal encoder (checkpointing is done at the chunk level in the encoder wrapper to avoid ~30% wasted backward compute from double-checkpointing).

---

Hardware & environment

Wall-clock time

Pretraining dominates the budget by >40×; finetuning is nearly free once a pretrained encoder exists (relevant for the seed-variance runs recommended in § Next steps — they reuse the same pretrained encoder and cost ~1 h each, not another 43 h).

---

Results

Headline (test set: PASTIS fold 4, 482 full 128×128 patches, 3,961,114 scored pixels, no TTA)

The primary number is the fixed-epoch-99 checkpoint (checkpoints/finetune/latest.tar), a deliberate protocol choice: best-checkpoint selection on only 152 validation crops was previously shown to manufacture a 147× variance artifact that vanished at fixed epoch, and the official reference implementation has no early stopping either. model_best.tar (epoch 37) is kept only as a reference point.

Comparison to baselines

The only like-for-like comparison is T-SRDA's own best-val checkpoint against A_linear at the same seed: 0.4433 vs. 0.4747 (Δ = −0.0314) — about 4.4× A_linear's own seed-to-seed standard deviation, so unlikely to be pure seed noise, but not quantifiable with a single T-SRDA seed.

Per-class IoU / F1 — epoch 99 (primary)

No dead classes — all 18 scored classes produce non-zero IoU. The four large, well-represented classes (Meadow, Corn, Soft winter wheat, Winter rapeseed) all land at 0.72–0.85 IoU; the deficit vs. baseline concentrates almost entirely in the confusable cereals (Spring barley, Mixed cereal, Winter triticale) — exactly the classes the baseline itself is least stable on across seeds.

---

Training dynamics

Pretraining — 100 epochs, 43.0 h — masked-reconstruction MSE

Converged cleanly and monotonically. No non-finite losses, no NaN-recovery events across all 198,400 optimizer steps.

Finetuning — 100 epochs, 60.3 min — validation mIoU

Validation mIoU peaks early (epoch 37, 0.4759) and then declines — consistent with overfitting to the 152 training crops. The last 20 epochs plateau at mean 0.4299 (sd 0.0156), and test mIoU tracks the same direction (0.4433 at ep37 → 0.4124 at ep99), though n=2 is too few points to draw a general conclusion from.

---

Repository contents

tsrda/
├── README.md                    this file
├── RESULTS.md                   full write-up this card is derived from
├── PATCH_model.md                notes on the 3-edit swap from DaViT → T-SRDA
├── config.py                     all hyperparameters, paths, PASTIS class table
├── model.py                      spatial encoder, T-SRDA wrapper, pretrain/finetune heads
├── temporal_srda.py              TemporalSRDAEncoder / TemporalSRDABlock
├── dataset.py                    PASTIS loading, cropping, collation
├── losses.py                     segmentation loss(es)
├── pretrain.py / finetune.py     training loops
├── evaluate.py                   test-set evaluation (mIoU/OA/F1/Kappa, per-class)
├── check_data.py / check_split.py / preflight.py   protocol/verification gates
├── smoke_model.py                shape + VRAM smoke test
├── run.sh / run_all.sh / launch.sh / status.sh      pipeline orchestration (tmux, resumable)
│
├── checkpoints/
│   ├── pretrain/                 checkpoint_{0,19,39,59,79,99}.tar (+ .utae.tar encoder-only)
│   │                              latest.tar / latest.utae.tar      (~476 MB total)
│   └── finetune/
│       ├── latest.tar             epoch 99 — PRIMARY reported checkpoint
│       └── model_best.tar         epoch 37 — best-val, reference only
│
├── runs/20260823_204044/         raw logs for this run
│   ├── preflight_log.txt          protocol gates
│   ├── check_data_log.txt         data assertions
│   ├── smoke_log.txt              VRAM + shape checks
│   ├── pretrain_log.txt           100 epochs
│   ├── finetune_log.txt           100 epochs, per-epoch metrics
│   ├── eval_ep99_PRIMARY_log.txt  the reported test numbers
│   └── eval_best_REFERENCE_log.txt
│
├── logs_prev_machine/            pre-port logs, kept for audit trail
└── run_all_console.txt           full console transcript of the run

Checkpoint files

---

Reproducing this run

Requires the PASTIS dataset locally (PASTIS_ROOT env var or a sibling PASTIS/ directory containing metadata.geojson).

git clone https://huggingface.co/Dhruv1000/TSRDA
cd TSRDA

# gates only
python3 preflight.py

# full pipeline (pretrain → finetune → eval), tmux-backed and resumable
bash launch.sh
bash status.sh          # check progress without attaching

# evaluate an existing checkpoint
python3 evaluate.py --ckpt checkpoints/finetune/latest.tar

# additional finetune seeds from the already-pretrained encoder (~1h each)
python3 finetune.py --pretrain_pth checkpoints/pretrain/checkpoint_99.utae.tar \
  --seed 42   --tag seed42
python3 finetune.py --pretrain_pth checkpoints/pretrain/checkpoint_99.utae.tar \
  --seed 1234 --tag seed1234

If evaluation OOMs on a smaller/shared GPU, rerun with EVAL_BATCH=2 (9.5 GB peak) or EVAL_BATCH=1 (5.05 GB peak) — metrics are numerically identical at any batch size (full-patch inference has no batch-dependent ops in eval mode).

Verification gates (all passed, re-runnable via preflight.py)

---

Honest summary & next steps

The port to the official protocol worked cleanly: fold-disjoint splits, exact step counts, a bit-identical pretext task, no dead classes, stable convergence, and a fixed-epoch primary metric that doesn't depend on validation noise. The measurement is sound.

The measurement says T-SRDA is 0.031 mIoU behind the simplest correct baseline at matched checkpoint selection and one seed — concentrated almost entirely in three cereal classes that the baseline itself is least stable on across seeds. This is a negative result, reported as measured, not a claim that the architecture is broken: it trains stably and lands in the same 0.41–0.48 mIoU band as every other arm of this project, it just doesn't win.

Ranked next steps (by information gained per GPU-hour, cheapest first):

1. Two more finetune seeds (42, 1234), ~2 GPU-hours — reuses the already-pretrained encoder. Without this, the −0.031 gap can't be told apart from a bad draw on unstable classes. Highest-value next run.
2. Remove the decoder confound, ~1 GPU-hour — finetune the same encoder with a plain linear head instead of the dual+boundary decoder.
3. Earlier-epoch encoders, ~1 GPU-hour each — milestones at epochs 19, 39, 59, 79 are already on disk; tests whether 43h of pretraining was even necessary.
4. Real date positions (USE_INDEX_POSITIONS=False) — another full 43h pretrain, so only worth it after 1–3 narrow things down. This is the one protocol choice that arguably handicaps T-SRDA specifically, since its design assumes date PE is available before KV reduction.

Citation

This repository ports and evaluates the SRDA attention mechanism from:

@article{zhou2026pdavit,
  title   = {PDAViT: Spatial-Reduction Dual-Attention Vision Transformer},
  author  = {Zhou, et al.},
  journal = {Neurocomputing},
  year    = {2026}
}

against the official STCLN PASTIS protocol and baselines:

@misc{stcln,
  title = {STCLN},
  howpublished = {\url{https://github.com/XiaoleiQinn/STCLN}}
}

on the PASTIS benchmark:

@inproceedings{garnot2021pastis,
  title     = {Panoptic Segmentation of Satellite Image Time Series with
               Convolutional Temporal Attention Networks},
  author    = {Sainte Fare Garnot, Vivien and Landrieu, Loic},
  booktitle = {ICCV},
  year      = {2021}
}