scene-atlas-small

Scene Atlas Small

An image–text embedding model for searching tabletop RPG art: battlemaps, scene and environment illustrations, portraits, tokens, items and maps. It is the lightweight model of Scene Atlas, a tool that helps Game Masters organise and search their art libraries, and its default on computers without a GPU.

It is laion/CLIP-ViT-B-32-laion2B-s34B-b79K fine-tuned on about 121,000 captioned TTRPG images. On a benchmark of Game Master search queries it scores nDCG@10 0.470, against 0.376 for CLIP ViT-B/32 LAION-2B (zero-shot, previous Scene Atlas small).

Variants

This repo holds the same model in two formats:

Use the quantised variant to run the model: it is what Scene Atlas downloads, it needs about 160 MB of memory, and it runs on CPU, CUDA and DirectML with ONNX Runtime 1.24 or later. Use the full variant to fine-tune further, or to work in PyTorch.

Evaluation

The benchmark has 664 search queries written the way a Game Master would type them, such as "medieval village", "burning airship" or "frozen tundra under the northern lights". Each query searches a gallery of 8,190 held-out test images, restricted to the query's art category. Test images were held out of training together with their known variants. A Claude Haiku judge graded the relevance of every image that any model below ranked in its top 10 for a query; only clear matches count as relevant.

The quantised variant was scored through ONNX Runtime with inputs fed exactly as Scene Atlas feeds them. Its difference from the full model is -0.59% (95% CI -2.05% to +0.85%, paired bootstrap over queries).

Usage

ONNX (quantised)

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from PIL import Image
from tokenizers import Tokenizer

d = snapshot_download("jennis0/scene-atlas-small", allow_patterns=["onnx/*"]) + "/onnx"
m = json.load(open(f"{d}/manifest.json"))
vision = ort.InferenceSession(f"{d}/{m['vision_encoder']}")
text = ort.InferenceSession(f"{d}/{m['text_encoder']}")

tok = Tokenizer.from_file(f"{d}/tokenizer/tokenizer.json")   # lowercases for you
tok.enable_truncation(max_length=m["tokenizer_max_length"])
tok.enable_padding(length=m["tokenizer_max_length"], pad_id=m["tokenizer_pad_id"])
ids = np.array([tok.encode("ruined dwarven forge with lava channels").ids], dtype=np.int64)
text_emb = text.run(["embeddings"], {"input_ids": ids, "attention_mask": np.ones_like(ids)})[0]

size = m["image_size"]
img = Image.open("map.jpg").convert("RGB")  # short side to `size` (bicubic), then centre crop
w, h = img.size
img = img.resize((max(size, round(w * size / h)), max(size, round(h * size / w))), Image.BICUBIC)
left, top = (img.width - size) // 2, (img.height - size) // 2
img = img.crop((left, top, left + size, top + size))
mean, std = np.array(m["image_mean"], np.float32), np.array(m["image_std"], np.float32)
x = ((np.asarray(img, np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None]  # float32 in, float32 out
image_emb = vision.run(["embeddings"], {"pixel_values": x})[0]

score = float(text_emb[0] @ image_emb[0])  # both outputs are L2-normalised

Transformers (full)

import torch
import torch.nn.functional as F
from PIL import Image
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("jennis0/scene-atlas-small").eval()
processor = AutoProcessor.from_pretrained("jennis0/scene-atlas-small")

text = processor(text=["ruined dwarven forge with lava channels"], padding=True, return_tensors="pt")
image = processor(images=[Image.open("map.jpg")], return_tensors="pt")
with torch.no_grad():
    t = model.get_text_features(**text)
    i = model.get_image_features(**image)
t, i = (F.normalize(getattr(x, "pooler_output", x), dim=-1) for x in (t, i))
score = (t @ i.T).item()

Quantisation

The ONNX variant is about 66% smaller than the fp16 checkpoint:

• Pruned vocabulary. The 49k-token vocabulary is cut to the 30,389 tokens that English text and TTRPG captions use, closed under BPE merges. A rare word splits into smaller kept pieces rather than becoming unknown. Token ids are unchanged, and the graph maps them to the compact table.
• 8-bit token table, with one scale per row.
• 4-bit weights in both encoders, quantised with GPTQ in activation order in blocks of 16. GPTQ was calibrated on captions and 1,024 thumbnails of training images only. Activations stay fp16, so GPU speed matches an fp16 model.

Spec: vocab=clip_wiki16k,table=int8,text=w4b16ao,vision=w4b16ao,ncal=1024, built with the onnx-export compress command of the Scene Atlas training repo.

Training

• Recipe: LoRA (rank 32 on the attention and MLP layers; 7.9M trainable parameters) with a softmax contrastive (InfoNCE) loss, at 256 pairs per step for 1,418 steps, then merged into the base weights. The full recipe is training/configs/openclip_vitb32_rc3.yaml.
• Data: 121,053 images, of which 32,838 tactical map, 30,264 portrait, 23,515 environment, 15,623 token, 13,806 scene, 3,777 item, 1,230 map. Captions and search-style descriptions were written by Claude Haiku and lowercased. Training mixed short and long captions, search strings and Game Master-style queries.

Limitations

• English only. Other languages still tokenise, as byte-level fallback pieces, but were never trained on.
• Fantasy and TTRPG art. Expect base-CLIP quality or worse on other domains.
• Relevance labels come from an LLM judge, checked against human judgments on a subset of queries. Absolute scores aren't comparable with human-judged benchmarks.

License

MIT, inherited from laion/CLIP-ViT-B-32-laion2B-s34B-b79K.