aloe-siglip2-base-in1k-lp

ALOE: Align Once to Explain

Using DINOv3? A newer ALOEv2 multi-resolution DINOv3 model is available and recommended — see rmaser/aloe-v2-dinov3-{small,base,large} (and the matching -in1k-lp classifier heads). ALOEv2 substantially improves dense correspondence and depth while keeping inherent B-cos explanations.

This repository contains one of the published ALOE vision backbones or ImageNet-1k linear-probe classifiers from the accepted CVPR 2026 poster "Align Once to Explain: Feature Alignment for Scalable B-cosification of Foundational Vision Transformers".

ALOE converts a frozen ViT-style foundation model into an inherently interpretable B-cos counterpart through a one-time, label-free feature-alignment stage. The aligned model is meant to be used as a drop-in visual backbone: it keeps strong downstream representations while exposing model-inherent B-cos explanations from the network itself.

ALOE stays within a fraction of a point of the original foundation models on ImageNet-1k linear-probe accuracy while lifting Grid-PG localization far above the teachers' best post-hoc explainers.

What ALOE Does

• Starts from a frozen teacher encoder such as supervised ViT-B/16, DINOv3, or SigLIP2.
• Builds a bias-free B-cos student with six-channel B-cos preprocessing and matching width, depth, and token routing.
• Aligns the student to the teacher on unlabeled image data using global embedding matching and token-level supervision at multiple depths.

The paper reports that this alignment is robust across supervised, self-supervised, and vision-language teachers, and is about 100-1000x more data-efficient than training B-cos models from scratch.

Published Checkpoints

The public ALOE backbone family includes:

The corresponding public ImageNet-1k linear-probe classifiers include the aligned backbone and its trained 1,000-class probe:

Main Metrics From The Paper

All numbers below are top-1 accuracy or GridPG localization scores in percent unless noted otherwise.

ImageNet-1k, Published Model Scale

Across every scale, ALOE holds ImageNet-1k linear-probe and k-NN accuracy within a fraction of a point of the teacher while lifting Grid-PG localization far above the teacher's strongest post-hoc explainer (AttnLRP). Per-checkpoint numbers are in the paper.

Explanations And Feature Geometry

ALOE B-cos attributions (middle) are object-centric and class-specific, while the PCA of the final image representation (bottom) shows the teacher's global feature geometry is preserved — semantics stay aligned while explainability improves.

Linear Evaluation On 10 Datasets, ViT-B/16

The 10-dataset protocol covers ImageNet-1k, Caltech101, Oxford Flowers, Food101, FGVC Aircraft, DTD, Stanford Cars, SUN397, CIFAR-10, and CIFAR-100.

Zero-Shot ImageNet-1k With SigLIP2 Text Encoder

Dense Prediction

For monocular depth with frozen ViT-B/16 features and a linear probe, ALOE improves over B-cosification on NYUv2-style depth metrics:

Usage

These models use custom ALOE Transformers code, so loading requires trust_remote_code=True.

from transformers import AutoImageProcessor, AutoModel

repo_id = "rmaser/aloe-dinov3-base"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

Hidden States Are Pre-Norm

With output_hidden_states=True, hidden_states[i] is the raw output of block i — in particular hidden_states[-1] is not last_hidden_state, which additionally passes through post_layernorm. This is the convention the distillation loss was defined against, and recent Transformers versions report the post-norm tensor in hidden_states[-1] on the official DINOv3 and ViT models, so the two APIs differ at that one index.

Use hidden_states[i] for anything layer-wise. These models supervise blocks n/3 - 1, 2n/3 - 1 and n - 1 of n, reaching 0.90–0.995 cosine similarity to the teacher's corresponding pre-norm features. Note that the deepest supervised block is n - 1, one short of the last one, and that post_layernorm was never part of the loss either — so last_hidden_state is only loosely aligned to the teacher's, and hidden_states[n - 1] is the deepest well-aligned representation. The ALOEv2 models (rmaser/aloe-v2-dinov3-*) supervise block n itself.

ImageNet-1k Class Explanations (-in1k-lp Only)

Use an -in1k-lp classifier checkpoint to obtain class-specific, model-inherent explanations directly through model.explain(...):

from PIL import Image
from transformers import AutoImageProcessor, AutoModelForImageClassification

repo_id = "rmaser/aloe-dinov3-base-in1k-lp"
processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageClassification.from_pretrained(repo_id, trust_remote_code=True)
model.eval()

image = Image.open("image.jpg").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values

result = model.explain(pixel_values, idx=None)
class_idx = int(result["explained_class_idx"][0])
print(f"Predicted ImageNet-1k class index: {class_idx}")

rgba = (result["explanation"][0] * 255).astype("uint8")
Image.fromarray(rgba).save("explanation.png")

idx=None explains the predicted class. Pass an ImageNet-1k class index to idx to explain a specific class instead. Do not wrap model.explain(...) in torch.inference_mode(): generating the attribution requires input gradients.

SigLIP2 Zero-Shot Explanations (Backbones Only)

For non--in1k-lp SigLIP2-family ALOE backbones, the aligned B-cos image encoder can be paired with the corresponding original SigLIP2 text encoder for zero-shot classification and zero-shot explanations.

import torch
import torch.nn.functional as F
from PIL import Image
from transformers import AutoImageProcessor, AutoModel, AutoTokenizer, Siglip2TextModel

repo_id = "rmaser/aloe-siglip2-base"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

processor = AutoImageProcessor.from_pretrained(repo_id, trust_remote_code=True)
image_model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).to(device)
image_model.eval()

text_model_id = image_model.config.aloe_base_model_name
tokenizer = AutoTokenizer.from_pretrained(text_model_id)
text_model = Siglip2TextModel.from_pretrained(text_model_id).to(device)
text_model.eval()

labels = ["a person eating spaghetti", "a person playing guitar", "a person running"]
prompts = [f"This is a photo of {label}.".lower() for label in labels]
tokens = tokenizer(
    prompts,
    padding="max_length",
    truncation=True,
    max_length=64,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    text_features = F.normalize(text_model(**tokens).pooler_output, dim=-1)

image = Image.open("image.jpg").convert("RGB")
pixel_values = processor(images=image, return_tensors="pt").pixel_values.to(device)

explanation = image_model.explain_language_features(
    pixel_values,
    text_features,
    idx=None,
)
# explanation["explanation"]         — RGBA attribution overlay, (1, H, W, 4)
# explanation["contribution_map"]    — input×gradient map, (1, 1, H, W)
# explanation["explained_class_idx"] — index into labels

idx=None explains the highest-scoring prompt; pass a label index to explain a specific prompt. Explanation calls currently expect one input image at a time.

Training Summary

• Alignment data: unlabeled web-scale images from CC3M, CC12M, or YFCC15M, with main results reported for YFCC15M.
• Default resolution: teacher default, typically 224 x 224.
• Objective: cosine alignment of global image features plus token-level features at roughly 1/3, 2/3, and final depth.
• Optimization: frozen teacher, AdamW, cosine learning-rate schedule, mixed precision, batch size 1024, random resized crop and horizontal flip.
• Evaluation: frozen-feature linear probing, weighted k-NN with k=20, SigLIP2 zero-shot transfer, dense linear probes, GridPG, pixel deletion, and human preference evaluation.

Citation

@inproceedings{maser2026align,
  title = {Align Once to Explain: Feature Alignment for Scalable B-cosification of Foundational Vision Transformers},
  author = {Maser, Raphael and Gairola, Siddhartha and Rao, Sukrut and Schiele, Bernt},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2026},
  note = {Poster}
}

Links

• Code: github.com/rmaser/ALOE
• Model collection: ALOE on Hugging Face
• Shared ALOE runtime code: rmaser/aloe-arch
• Paper: Align Once to Explain (CVPR 2026)

License

This model is released under the Creative Commons Attribution-NonCommercial ShareAlike 4.0 International License. The methods described in this work are patent pending.

Limitations

ALOE is designed to make ViT-style visual backbones inherently interpretable through B-cos explanations. It does not remove the usual limitations of the teacher models, the unlabeled alignment data, ImageNet-1k classifiers, or downstream datasets. The current public checkpoints should be treated as research artifacts.