spatial-vlm-adapters

Lightweight Visual-Token Spatial Adapters for Frozen VLMs

Tiny adapters that improve spatial reasoning in frozen vision–language models by injecting at the visual-token interface (projector/merger output + early decoder layers). The base VLM is never fine-tuned; only these adapters (0.003–0.08% of the model) are trained, on VSR.

They accompany our analysis that VLM spatial failure is mis-routing / a decision-stage language prior, not missing perception: the spatial relation is already decodable in the visual tokens from layer 0, but a language prior over relation words buries the correct answer (e.g. "under" and "behind" are emitted 0% of the time). Injecting a small learned signal at the visual-token interface — the site we localize causally with activation patching — unlocks it.

Checkpoints

Each *.pt is a state_dict of a bottleneck adapter Δ(x) = W_up · GELU(W_down · x) added to the visual tokens (and, for the Qwen files, a few early decoder layers at image positions).

All numbers are VSR test accuracy gains with the base VLM frozen. The adapter beats a well-tuned LoRA (attention weights) at 11–55× fewer parameters, and matches its cross-benchmark transfer.

Boundaries (reported honestly)

• The gain is largely a learned de-biasing steer at the visual tokens; on visually homogeneous data a constant, content-free offset reproduces most of it. A training-free alternative (prior-calibrated / contrastive decoding) recovers the same failing poles with zero parameters — use that first.
• Transfer is relation-type-specific: helps location/proximity, can hurt orientation/viewpoint.
• The interface adapter does not work on Perceiver-resampler connectors (Idefics2).

Usage sketch

import torch, torch.nn.functional as F
from torch import nn

class Bottleneck(nn.Module):
    def __init__(self, d, r):
        super().__init__()
        self.down = nn.Linear(d, r); self.up = nn.Linear(r, d)
    def forward(self, x):
        return self.up(F.gelu(self.down(x)))

sd = torch.load("qwen3vl-8b_adapter_rank8.pt")   # {"merger": ..., "L2": ..., "L6": ..., "L10": ...}
# register forward hooks that add Bottleneck(x) to the projector output (key "merger")
# and to image-token positions at decoder layers {2,6,10} (keys "L2","L6","L10").

See the project repository for the exact hook/registration code and evaluation scripts.