scene-atlas-medium

Scene Atlas Medium

An image–text embedding model for searching tabletop RPG art: battlemaps, scene and environment illustrations, portraits, tokens, items and maps. It is the default model of Scene Atlas, a tool that helps Game Masters organise and search their art libraries.

It is google/siglip2-base-patch16-384 fine-tuned on about 121,000 captioned TTRPG images. On a benchmark of Game Master search queries it scores nDCG@10 0.607, against 0.540 for SigLIP2 base-384 (zero-shot) and 0.505 for SigLIP2 base-224 (zero-shot, previous Scene Atlas medium).

Variants

This repo holds the same model in two formats:

Use the quantised variant to run the model: it is what Scene Atlas downloads, it needs about 235 MB of memory, and it runs on CPU, CUDA and DirectML with ONNX Runtime 1.24 or later. Use the full variant to fine-tune further, or to work in PyTorch.

Evaluation

The benchmark has 664 search queries written the way a Game Master would type them, such as "medieval village", "burning airship" or "frozen tundra under the northern lights". Each query searches a gallery of 8,190 held-out test images, restricted to the query's art category. Test images were held out of training together with their known variants. A Claude Haiku judge graded the relevance of every image that any model below ranked in its top 10 for a query; only clear matches count as relevant.

The quantised variant was scored through ONNX Runtime with inputs fed exactly as Scene Atlas feeds them. Its difference from the full model is -0.43% (95% CI -1.58% to +0.71%, paired bootstrap over queries).

Usage

ONNX (quantised)

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from PIL import Image
from tokenizers import Tokenizer

d = snapshot_download("jennis0/scene-atlas-medium", allow_patterns=["onnx/*"]) + "/onnx"
m = json.load(open(f"{d}/manifest.json"))
vision = ort.InferenceSession(f"{d}/{m['vision_encoder']}")
text = ort.InferenceSession(f"{d}/{m['text_encoder']}")

tok = Tokenizer.from_file(f"{d}/tokenizer/tokenizer.json")   # lowercases for you
tok.enable_truncation(max_length=m["tokenizer_max_length"])
tok.enable_padding(length=m["tokenizer_max_length"], pad_id=m["tokenizer_pad_id"])
ids = np.array([tok.encode("ruined dwarven forge with lava channels").ids], dtype=np.int64)
text_emb = text.run(["embeddings"], {"input_ids": ids, "attention_mask": np.ones_like(ids)})[0]

size = m["image_size"]  # plain resize to a square, no crop
img = Image.open("map.jpg").convert("RGB").resize((size, size), Image.BILINEAR)
x = ((np.asarray(img, np.float32) / 255 - 0.5) / 0.5).transpose(2, 0, 1)[None].astype(np.float16)
image_emb = vision.run(["embeddings"], {"pixel_values": x})[0]

score = float(text_emb[0] @ image_emb[0])  # both outputs are L2-normalised

Transformers (full)

import torch
import torch.nn.functional as F
from PIL import Image
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("jennis0/scene-atlas-medium").eval()
processor = AutoProcessor.from_pretrained("jennis0/scene-atlas-medium")

# Pad text to 64 tokens and pass no attention mask: SigLIP2 is trained that way.
text = processor(text=["ruined dwarven forge with lava channels"],
                 padding="max_length", max_length=64, return_tensors="pt")
image = processor(images=[Image.open("map.jpg")], return_tensors="pt")
with torch.no_grad():
    t = model.get_text_features(input_ids=text["input_ids"])
    i = model.get_image_features(pixel_values=image["pixel_values"])
t, i = (F.normalize(getattr(x, "pooler_output", x), dim=-1) for x in (t, i))
score = (t @ i.T).item()

Two things that break SigLIP2 text embeddings:

• A padding attention mask. SigLIP2 attends over all 64 positions and pools the last one, which is padding. A padding mask changes that and collapses retrieval: nDCG@10 falls to about 0.03, for this model and for the base model. The ONNX text graph ignores its attention_mask input, so callers that pass one are safe.
• Capital letters. The model was trained on lowercased text. Both tokenizers in this repo lowercase automatically. If you tokenise some other way, lowercase first.

Quantisation

The ONNX variant is about 79% smaller than the fp16 checkpoint:

• Pruned vocabulary. The 256k-token multilingual vocabulary is cut to the 52,113 tokens that English text and TTRPG captions use, closed under BPE merges. A rare word splits into smaller kept pieces rather than becoming unknown. Token ids are unchanged, and the graph maps them to the compact table.
• 8-bit token table, with one scale per row.
• 4-bit weights in both encoders, quantised with GPTQ in blocks of 32 (vision) and 16 (text). GPTQ was calibrated on captions and thumbnails of training images only. Activations stay fp16, so GPU speed matches an fp16 model.

Spec: vocab=wiki32k,table=int8,text=w4b16ag,vision=w4b32ag, built with the onnx-export compress command of the Scene Atlas training repo.

Training

• Recipe: LoRA (rank 32 on the attention and MLP layers; 10.9M trainable parameters) with a sigmoid contrastive loss, at 128 pairs per step for 1,891 steps, then merged into the base weights. The full recipe is training/configs/siglip2_base_384_rc4.yaml.
• Data: 121,051 images, of which 32,837 tactical map, 30,264 portrait, 23,515 environment, 15,623 token, 13,806 scene, 3,777 item, 1,229 map. Captions and search-style descriptions were written by Claude Haiku and lowercased. Training mixed short and long captions, search strings and Game Master-style queries.

Limitations

• English only. Other languages still tokenise, as byte-level fallback pieces, but were never trained on.
• Fantasy and TTRPG art. Expect base-SigLIP2 quality or worse on other domains.
• Relevance labels come from an LLM judge, checked against human judgments on a subset of queries. Absolute scores aren't comparable with human-judged benchmarks.

License

Apache 2.0, inherited from google/siglip2-base-patch16-384.