GPN

Glimpse Prediction Networks (GPN)

This repo contains the best pretrained GPN models (B, S, R and RS variants), based on the SimCLR backbone, trained on DeepGaze3 glimpses on COCO scenes.

For replicating the analyses reported in the paper (https://arxiv.org/abs/2511.12715), check the github repository: https://github.com/KietzmannLab/GPN

Usage

First setup the glimpse encoder backbone and the GPN:

from huggingface_hub import snapshot_download
import torch
import sys, os
from torchvision.models import resnet50
from torchvision import transforms

REPO_ID = "novelmartis/GPN"

# download whole repo (code + weights) to a local folder

repo_path = snapshot_download(REPO_ID)

# make repo importable

sys.path.insert(0, repo_path)

from gpn_model import build_gpn
device = torch.device("cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu")

# setup glimpse encoder - SimCLR-RN50

# obtained from: https://huggingface.co/lightly-ai/simclrv1-imagenet1k-resnet50-1x
ckpt = os.path.join(repo_path, "SimCLR ResNet50 1x.pth")

encoder = resnet50(weights=None)
encoder.load_state_dict(torch.load(ckpt, map_location="cpu")['state_dict'])
preprocess = transforms.Compose([
    transforms.Resize(224, antialias=True),
    transforms.ConvertImageDtype(torch.float),
])
encoder.to(device)
encoder.eval()

# setting up avgpool activation extraction from the RN50-SimCLR

activation = {} # remember to empty the list after each forward pass
def get_activation(name): # setting hooks
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

encoder.avgpool.register_forward_hook(get_activation('avgpool'))

# choose GPN variants B/S/R/RS

variant = "R"
ckpt = os.path.join(repo_path, f"gpn_{variant}.pth")

gpn = build_gpn(variant=variant)
gpn.load_state_dict(torch.load(ckpt, map_location="cpu"))
gpn.to(device)
gpn.eval()

Next, gather your N glimpse images for each of the B scenes into a [B,N,H,W,3] uint8 numpy array.

Note that during training, COCO scene images were scaled on their lower side to 256px and a central 256px crop was taken. DeepGaze3 was fed this image to obtain fixations. 91px crops were taken around the fixations as glimpses. I'd advise you to do similar preprocessing steps on your images to obtain 'input_glimpses', unless you have apriori reasons to do otherwise.

Gather the glimpse embeddings:

# obtain 'glimpse_embeddings' [B,N,2048]

input_glimpses_tensor = preprocess(torch.from_numpy(input_glimpses.reshape(B*N,H,W,3)).permute(0, 3, 1, 2)).to(device) 
with torch.no_grad():
    _ = encoder(input_glimpses_tensor)
    glimpse_embeddings = activation['avgpool'].detach().cpu().numpy().squeeze()
glimpse_embeddings = glimpse_embeddings.reshape(B,N,-1)

Gather the saccade_inputs [B,N,2], where dim=0 of a saccades (x-coordinate) is positive when the saccade points to the right, and dim=1 of a saccade (y-coordinate) is positive when the saccade points to the bottom. Consider that the GPNs were trained on 256px scenes which bound the distribution of saccades—please scale your saccade vectors accordingly. Finally, pass the glimpse embeddings into the GPN, alongwith the saccades:

# run the gpn on glimpse_embeddings and saccade_inputs
# set return_all_actvs=True to acquire internal representations

representations, predictions = gpn(torch.from_numpy(glimpse_embeddings).to(device), torch.from_numpy(saccade_inputs).to(device), return_all_actvs=True)

GPN outputs:

• predictions[0]: [B,N,2048] contain the prediction of the T+1 glimpse embedding at each T
• representations is a dict; e.g. representations['lstm_out'], when variant = "R", have the best alignment to ventral and parietal ROI RDMs in NSD.

Using the GSNs

GSNs have the same architecture as GPNs but are trained with different objectives. The data preprocessing looks exactly as the same as for the GPNs. The main difference is the readout. Use predictions[1] for sGSN (semantic loss) or predictions[2] for cGSN (multiclass loss). See the paper for details regarding these objectives.

The GSNs available here were trained to match the architecture of GPN-R-SimCLR.

In terms of choosing the GSNs, instead of the "choose GPN variants" block above use:

# choose GSN variant c/s

variant = "s"
ckpt = os.path.join(repo_path, f"{variant}gsn_R.pth")

gpn = build_gsn(variant=variant)
gpn.load_state_dict(torch.load(ckpt, map_location="cpu"))
gpn.to(device)
gpn.eval()

and use the rest of the code as-is (while being aware that you're indeed now operating a GSN and not a GPN).