astroclip

AstroCLIP

AstroCLIP is a dual encoder for galaxy data. It maps galaxy image cutouts and galaxy spectra into a shared embedding space using a DINOv2-style image encoder, a SpecFormer spectrum encoder, cross-attention pooling heads and a symmetric contrastive loss.

This repository packages the published AstroCLIP checkpoint in a Transformers-compatible remote-code format. It can be loaded with the normal Auto classes by passing trust_remote_code=True.

Usage

import numpy as np

from transformers import AutoModel, AutoProcessor


model = AutoModel.from_pretrained("giovannicozzolongo/astroclip", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("giovannicozzolongo/astroclip", trust_remote_code=True)

image = np.load("galaxy_grz_cutout.npy")  # shape: (3, height, width), bands: g, r, z
spectrum = np.load("galaxy_spectrum.npy")

inputs = processor(images=image, spectra=spectrum, return_tensors="pt")
outputs = model(**inputs)

image_embeddings = outputs.image_embeds
spectrum_embeddings = outputs.spectrum_embeds
similarity = outputs.logits_per_image

For cutouts with additional bands, construct AstroClipImageProcessor with band_indices so the selected channels correspond to the g,r,z bands used by the upstream checkpoint.

Inputs

Image input:

• channel-first or channel-last image arrays
• expected bands: g,r,z
• centre-cropped to 144x144 pixels
• converted with the AstroCLIP Legacy Survey arcsinh RGB transform

Spectrum input:

• raw single-channel spectra
• accepted shapes: (length,), (batch, length) or (batch, length, 1)
• standardised per sample, sliced into overlapping sections and padded to the SpecFormer feature format

For the published checkpoint, a 7781-point spectrum is converted to spectrum_values with shape (batch, 778, 22).

Validation

The checkpoint conversion maps the upstream Lightning checkpoint into Transformers-style weights and saves them as safetensors.

Conversion checks:

• AstroCLIP checkpoint: 446 mapped source keys, 0 skipped keys
• saved model reloads with AutoModel.from_pretrained(..., trust_remote_code=True)
• saved processor reloads with AutoProcessor.from_pretrained(..., trust_remote_code=True)
• synthetic image+spectrum forward pass returns (batch, 1024) image and spectrum embeddings

Fixed-input parity against the upstream implementation passed with rtol=1e-4, atol=1e-5 for image embeddings, spectrum embeddings, logits and contrastive loss.

Intended Use

This model is intended for research workflows involving galaxy images and spectra, including:

• cross-modal retrieval between images and spectra
• embedding extraction for downstream astronomy tasks
• reproducible experiments based on the published AstroCLIP checkpoint

It is not intended for clinical, safety-critical or consumer decision systems.

Training Data

The upstream AstroCLIP paper describes training on paired galaxy images and spectra. The image side uses Legacy Survey g,r,z cutouts and the spectrum side uses DESI spectra. See the paper and upstream repository for dataset details.

Limitations

• AstroCLIP is designed for galaxy image/spectrum data.
• Performance outside the upstream data distribution is not established here.
• Processor behaviour depends on the upstream image transform and spectrum slicing convention.
• Loading requires trust_remote_code=True because the model code is included in this Hub repository rather than in the main Transformers library.

References

• Paper: https://arxiv.org/abs/2310.03024
• Upstream code: https://github.com/PolymathicAI/AstroCLIP
• Upstream checkpoint: https://huggingface.co/polymathic-ai/astroclip

@misc{parker2024astroclipcrossmodalfoundationmodel,
      title={AstroCLIP: A Cross-Modal Foundation Model for Galaxies},
      author={Liam Parker and Francois Lanusse and Siavash Golkar and
              Leopoldo Sarra and Miles Cranmer and Alberto Bietti and
              Michael Eickenberg and Geraud Krawezik and Michael McCabe and
              Ruben Ohana and Mariel Pettee and Bruno Regaldo-Saint Blancard and
              Tiberiu Tesileanu and Kyunghyun Cho and Shirley Ho},
      year={2024},
      eprint={2310.03024},
      archivePrefix={arXiv},
      primaryClass={astro-ph.IM},
      doi={10.1093/mnras/stae1450},
      url={https://arxiv.org/abs/2310.03024},
}