rs

RS: extreme image tokenization for satellite→ground vision-language inference

Two models for the same problem — get a remote-sensing image from orbit to a VLM on the ground over a link that cannot carry pixels.

Both build on RSThinker (GLM-4.1V-9B + Geo-CoT) and a One-D-Piece tokenizer fine-tuned on grayscale remote-sensing imagery. RSThinker itself is not redistributed here — point model_name_or_path at the upstream weights.

                  72 bytes / 256×256 crop
  ┌───────────┐   ────────────────────►   ┌──────────────────────┐
  │ satellite │      48 × 12-bit codes    │ ground               │
  │ 5.92 M    │                           │ decoder + GLM-4.1V   │
  └───────────┘                           └──────────────────────┘

---

satlink_pixel/ — the pixel path (deployed, benchmarked)

The satellite runs encoder + VQ only (5.92 M params). The ground decodes the codes back to pixels inside the VLM and answers.

Wire format. Everything fixed at deployment — frame size, tile size, tokens per crop, codebook width — is a configured constant, not a transmitted field. Tile count per frame is then constant too, so the tile index and the frame boundary are implied by position in the stream: there is no header. 48 tokens × log2(4096) bits = 72 bytes per 256×256 crop, 910× smaller than the raw grayscale it stands for, 0% protocol overhead.

n_tok is a rate knob. One-D-Piece is trained with tail-token drop, so tokens are ordered by importance (measured entropy 10.2–10.9 bits at positions 0–7, 11.1–11.7 at 40–47) and any prefix is a valid encoding. Cutting it trades detail for bandwidth linearly, at inference time.

The decoder is a submodule of the VLM, not a preprocessing step. Codes → latents → pixels → patches → vision tower → LLM happens in one forward, one checkpoint, one graph.

from satlink.protocol import LinkProfile, unpack_frame
from satlink.sat.encoder import SatEncoder
from satlink.frame import VQFrame
from satlink.ground.engine.llm import LLM

prof = LinkProfile(512, 512)                        # agreed by both ends, never transmitted
sat  = SatEncoder("satlink/weights/sat_encoder.pt", prof, device="cuda")

frame = sat.encode(image)                           # structured: a VQFrame
wire  = sat.encode_wire(image)                      # packed: 288 bytes for a 512×512 frame

llm = LLM(model="<RSThinker>", profile=prof, decoder="satlink/weights/ground_decoder.pt")
out = llm.generate([prompt_ids], [params], frames=[frame])

satlink/ground/infer.py is the reference inference script (cache-free, deliberately simple). satlink/ground/engine/ is a vendored nano-vLLM (MIT) with paged KV, CUDA graphs and continuous batching.

Measured

VRSBench, identical items, IoU@0.5 / accuracy:

The split costs nothing measurable. On original (uncompressed) images the same model scores 79.5% on referring — the ~35-point gap is the 48-token bottleneck, not the satellite/ground split.

Verified, not asserted

A reimplementation that loads and produces plausible output is worth nothing; a mis-ported rope degrades quality without erroring. satlink/bench/ checks:

test_isolation.py is what makes "the halves are decoupled" checkable rather than aspirational: the satellite package cannot import decoder code, and satlink/sat/opd_encoder.py is truncated before TiTokDecoder so it physically cannot.

---

latent_sft_dec12/ — the latent path (trained and benchmarked)

Skips pixel reconstruction entirely. The One-D-Piece decoder is cut off at layer 6, where the representation is already a 16x16 grid per crop, and projected straight into the vision tower's patch-embedding space. The satellite bundle is byte-identical between the two paths — both send the same codes; only the ground tokenizer differs, and it shrinks from 337.8M to 76.2M.

codes → 6 of 24 decoder layers (frozen) → cross-crop interaction + projection (TRAINED)
      → GLM-4.1V vision tower (frozen) → LLM (LoRA)

The hard part is not the projection. A frame is many crops; they overlap by an amount that depends on the frame size; their origins are not multiples of the 16-pixel feature cell; and the vision tower wants ONE patch grid at 14 px/patch in a merge-block-major order that is not raster. The bridge resamples each crop's feature grid at the target patch centres and averages where crops overlap. test_bridge_geometry.py checks placement, order and coordinates: 0 misplacements over 10 892 patches across five frame sizes including overlapping and non-square.

Cross-crop information travels two ways: attention over the assembled grid, and a whole-image context vector modulating the scale and shift of both norms in every block.

Trained: 149.5M of 10.8B (interaction 50.4M + projection 3.9M + LoRA 95.2M). Frozen: encoder, VQ, the decoder layers, the entire vision tower. 1 epoch over GeoCoT380k, loss 1.095 → 0.817.

Measured — same items, same engine, same scorer

(n = 1000 / 996 / 690. base on original images scores 79.5 referring and 69.4 LHRS — the gap to any row above is the 48-token bottleneck itself, not the satellite/ground split, which costs nothing measurable: pixel path vs base is +0.7, p = 0.47.)

Not "worse" or "better" — coarse versus fine. Going from 6 to 12 decoder layers (with 3x the training) closed the mIoU gap to noise: the boxes are as good on average. Acc@0.75 barely moved (12.6 → 13.3 against the pixel path's 20.8). That is a resolution ceiling: the bridge upsamples a 16x16 per-crop feature grid to a 36x36 patch grid, and upsampling does not create detail. More decoder depth improves feature quality, not spatial resolution. VQA and LHRS, which do not need sub-patch precision, are unaffected or better.

The comparison is not matched. The latent models had SFT on GeoCoT380k; the pixel path is the base model run zero-shot on reconstructions. "Latent is better at VQA" and "latent was trained more" are not separated by these runs. The missing control is an SFT'd pixel path.

The +32.5 on VQA colour questions is almost certainly not colour perception. The tokenizer is grayscale — both paths get the same codes and neither carries colour. SFT most likely fitted this dataset's answer distribution for colour questions.

Scoring

Open-ended VQA is scored by bench_score.vqa_ok, not string equality. Strict equality marked aerial wrong against aerial view on ~4% of rows of every path; naive substring matching is worse, scoring 1 correct for 10 and No correct for north-south. The rule is: numbers must match numerically and exactly; otherwise one side's whole content words must be a subset of the other's. Correcting it moved all three paths by ~+3.2 and left their ordering unchanged.

Precomputing codes

The encoder is frozen, so encoding per epoch is waste. precompute_codes.py does it once: 60 GB of JPEG-in-parquet → 1.9 GB of codes, a 210 GB datasets Arrow cache goes with it, and LLaMA-Factory's ~23-minute "Converting format of dataset" pass becomes seconds. Stores the full 256 tokens per crop; training slices the first 48, so n_tok stays free.

Note: VQ argmin flips ~3% of codes under different batch shapes (4096 entries, many near-ties), so codes are not bit-reproducible across batch configurations. Feeding identical codes through both paths agrees to 1.0e-06, so this is quantiser jitter, not a plumbing difference.

Using it

The halves run as separate programs and talk through the wire format. That is the deployment shape, not a demo convention, and it is why each half can be validated alone.

# SATELLITE -- 5.9M params; imports satlink.sat + satlink.protocol and nothing else
python -m satlink.examples.sat_encode \
    --image satlink/assets/grid3x3_600x600.png --out codes.json
# 600x600 -> 3x3 crops (84 px overlap) -> 648 bytes, 556x vs raw grayscale, 411 ms

# GROUND -- never imports satellite code; a codes file is enough
python -m satlink.examples.ground_answer --wire codes.json --model <RSThinker> \
    --latent-ground satlink/weights/latent_ground_dec12.pt --lora latent_sft_dec12 \
    --question "where is the ship"

--out codes.json writes the codes as readable text; any other suffix writes the packed bytes a radio would carry. Swap --latent-ground/--lora for --decoder satlink/weights/ground_decoder.pt to run the pixel path on the same codes — the satellite bundle is identical for both.

Tests

Split the same way as the code, so neither half needs the other:

python -m satlink.bench.test_protocol   # shared contract; no torch, no weights  (~1 s)
python -m satlink.bench.test_sat        # satellite only; --no-model for geometry alone
python -m satlink.bench.test_ground --decoder ... --latent-ground ... --model ...
python -m satlink.bench.test_isolation  # neither package can reach outside itself

Five images ship in satlink/assets/ covering the cases that break tiling — exact multiples of the tile size (zero overlap), 84 px and 75 px overlap, non-square, and a single crop — with reference.json recording their codes, reconstructions, and a reference answer.

What is judged, and what is only reported. Codes are not required to match bitwise: the VQ step is an argmin over 4096 near-tied entries, and fp16 flips ~0.5% of them against an fp32 reference. Measured, that costs 0.47/255 mean pixel difference against a reconstruction error of 10.25/255, and the differences sit on high-frequency edges where moving an edge by a fraction of a pixel swings one pixel by 159/255 without changing anything semantic. So:

Free generation is reported, not judged, on purpose: greedy decoding turns any near-tie into a different trajectory, so two healthy devices can diverge at token 43 and never re-converge. (One of the five bundled images does exactly that, at 98.4% teacher-forced top-1 agreement.) Teacher forcing is the stable measurement — one forward pass, no decoding, a number that moves smoothly with numerical drift.

Batch shape does not affect codes: fp32 at chunk 8 and chunk 64 agree bitwise, with zero pixel difference. Precision does.

The heavier parity suite (test_vision_parity, test_text_parity, test_stack_parity, test_engine_parity) needs the 10B checkpoint and several minutes.

Layout

satlink_pixel/satlink/
  protocol.py      wire contract, zero dependencies
  frame.py         VQFrame — the structured form, shared by both ends
  sat/             satellite package — torch only, no decoder code
  ground/          decoder + GLM-4.1V + vendored nano-vLLM engine
  weights/         sat_encoder.pt (23 MB) · ground_decoder.pt (1.3 GB, pixel path)
                   latent_ground.pt (589 MB, 6 layers) ·
                   latent_ground_dec12.pt (891 MB, 12 layers -- the better latent model)
  ground/latent_bridge.py · modeling_latentvlm.py   the latent path
  examples/        sat_encode.py · ground_answer.py -- the halves, run separately
  assets/          five test images + reference.json (codes, reconstructions, answers)
  bench/           throughput, isolation, and the parity suite
  eval/            eval_satlink.py serves both paths (--latent-ground), 8-way shardable

latent_sft/         6 decoder layers, 1 epoch
latent_sft_dec12/   12 decoder layers, 3 epochs -- use this one
  adapter_model.safetensors · opd_bridge.pt · code/

Licence

Apache-2.0. Vendored One-D-Piece (Apache-2.0, ByteDance) and nano-vLLM (MIT) keep their notices.