AnyUp-transformers

AnyUp Transformers Models

Self-contained Hugging Face checkpoints converted from the official AnyUp weights (anyup_paper.pth, anyup_multi_backbone.pth).

Each subfolder is a standalone Transformers repo (config.json, model.safetensors, image processor, pipeline, remote code) and loads with trust_remote_code=True.

Original paper: AnyUp: Universal Feature Upsampling (ICLR 2026 Oral)
Original code: wimmerth/anyup
Converted for Hugging Face by: BiliSakura
License (weights / original code): CC BY 4.0

What AnyUp does

AnyUp upsamples features from any vision encoder (DINO, CLIP, SAM, ResNet, …) to a higher spatial resolution, guided by a high-resolution RGB image. It is not encoder-specific: the same weights work at arbitrary feature channel counts and resolutions.

Inputs:

1. a high-resolution RGB guide (ImageNet mean/std)
2. a low-resolution feature map (B, C, h, w)

Output: upsampled features (B, C, H, W).

Checkpoints

Released weights use windowed multi-head cross-attention. After loading, call model.enable_natten() or AnyUpModel.from_pretrained(..., use_natten=True) to swap in the optional NATTEN kernel (copies Q/K from the MHA checkpoint; windows differ slightly from the paper).

Usage

from transformers import pipeline
from PIL import Image
import torch

MODEL = "/path/to/AnyUp-transformers/anyup-multi-backbone"

pipe = pipeline(
    task="image-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
)

image = Image.open("guide.png").convert("RGB")
lr_features = torch.randn(1024, 16, 16)  # any encoder, any C
hr_features = pipe(image, features=lr_features, return_tensors=True)
print(hr_features.shape)  # (1, 1024, H, W)  — H,W match the guide

Custom output size (for example patch-14 of a 2016 px tile → 144×144):

hr_features = pipe(
    image,
    features=lr_features,
    output_size=(144, 144),
    q_chunk_size=10,
    return_tensors=True,
)

The task is also inferred if you omit it (this repo registers a single custom pipeline):

pipe = pipeline(model=MODEL, trust_remote_code=True)

Load components directly:

from transformers import AutoImageProcessor, AutoModel

processor = AutoImageProcessor.from_pretrained(MODEL, trust_remote_code=True)
model = AutoModel.from_pretrained(MODEL, trust_remote_code=True)

inputs = processor(images=image, return_tensors="pt")
outputs = model(
    pixel_values=inputs["pixel_values"],
    features=lr_features.unsqueeze(0),  # (B, C, h, w)
    output_size=(144, 144),
)
upsampled = outputs.last_hidden_state  # (B, C, 144, 144)

Already ImageNet-normalized tensors can skip the processor and go straight to the model or pipeline:

guide = torch.randn(1, 3, 224, 224)          # ImageNet-normalized
feats = torch.randn(1, 1024, 16, 16)
upsampled = pipe(guide, features=feats, return_tensors=True)

Preprocessing

preprocessor_config.json matches the official inference recipe:

Notes

• Checkpoint keys are kept identical to the official nn.Module tree (image_encoder.*, cross_decode.*, rope.freqs, …), so conversion is a strict load_state_dict.
• q_chunk_size trades speed for memory. For MHA it is query tokens; for NATTEN it is low-res rows, not query tokens.
• NATTEN is optional and not stored in model.safetensors. Enabling it requires a NATTEN build matching your CUDA / PyTorch versions.

Citation

@inproceedings{wimmer2026anyup,
    title={AnyUp: Universal Feature Upsampling},
    author={Wimmer, Thomas and Truong, Prune and Rakotosaona, Marie-Julie and Oechsle, Michael and Tombari, Federico and Schiele, Bernt and Lenssen, Jan Eric},
    booktitle={Proceedings of the International Conference on Learning Representations ({ICLR})},
    year={2026}
}