relsim-anonymous-caption-qwen25vl-lora

Relational Visual Similarity --- Anonymous Captioning Model

<p align="left"> <a href="https://arxiv.org/abs/2512.07833"><img src="https://img.shields.io/badge/arXiv-Paper-b31b1b?logo=arxiv&logoColor=white" alt="arXiv"></a> <a href="#citation"><img src="https://img.shields.io/badge/BibTeX-Citation-blue" alt="BibTeX"></a> <a href="https://huggingface.co/datasets/thaoshibe/anonymous-captions-114k"><img src="https://img.shields.io/badge/🤗-Dataset-yellow" alt="HuggingFace Dataset"></a> <a href="https://thaoshibe.github.io/relsim/"><img src="https://img.shields.io/badge/ProjectPage-up-green" alt="Project Page"></a> <a href="https://thaoshibe.github.io/relsim/data_viewer/index.html"><img src="https://img.shields.io/badge/Dataset-Viewer-green" alt="Dataset Live view"></a> <a href="https://thaoshibe.github.io/relsim/retrieve/index.html"><img src="https://img.shields.io/badge/ImageRetrieval-Results-green" alt="Image Retrieval Results"></a> <a href="https://github.com/thaoshibe/relsim/stargazers"><img src="https://img.shields.io/github/stars/thaoshibe/relsim?style=social" alt="GitHub stars"></a> </p> <!-- | *We introduce a new visual similarity notion — relational visual similarity (relsim) — which captures the internal relational logic of a scene rather than its surface appearance.* | --> <tr> <!-- <td><img src="https://raw.githubusercontent.com/thaoshibe/thaoshibe.github.io/main/images/relsim.gif" height="350px"></td> --> <td><a href="https://thaoshibe.github.io/relsim/"><b>Relational Visual Similarity</b></a> (arXiv 2025) <br> <a href="https://thaoshibe.github.io/">Thao Nguyen</a>¹, <a href="https://sichengmo.github.io/">Sicheng Mo</a>³, <a href="https://krsingh.cs.ucdavis.edu/">Krishna Kumar Singh</a>², <a href="https://yilinwang.org/">Yilin Wang</a>², <a href="https://jshi31.github.io/jingshi/">Jing Shi</a>², <a href="https://scholar.google.com/citations?user=MqWYTj0AAAAJ&hl=en">Nicholas Kolkin</a>², <a href="https://scholar.google.com/citations?user=B_FTboQAAAAJ&hl=en">Eli Shechtman</a>², <a href="https://pages.cs.wisc.edu/~yongjaelee/">Yong Jae Lee</a>^{1,2, ★}, <a href="https://yuheng-li.github.io/">Yuheng Li</a>^{1, ★} <br>(★ Equal advising) <br> 1- University of Wisconsin–Madison; 2- Adobe Research; 3- UCLA</td> </tr> </table>

TL;DR: We introduce a new visual similarity notion: relational visual similarity, which complements traditional attribute-based perceptual similarity (e.g., LPIPS, CLIP, DINO).

<details> <summary> Click here to read <b>Abstract 📝</b></summary> Humans do not just see attribute similarity---we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach’s skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image–caption dataset in which the captions are anonymized---describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision–Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it---revealing a critical gap in visual computing. </details>

---

Anonymous captions are image captions that do not refer to specific visible objects but instead capture the relational logic conveyed by the image.

You can use this anonymous captioning model as below:

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel
from qwen_vl_utils import process_vision_info
from PIL import Image
import requests

# Load model from HuggingFace
base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "thaoshibe/relsim-anonymous-caption-qwen25vl-lora")
processor = AutoProcessor.from_pretrained("thaoshibe/relsim-anonymous-caption-qwen25vl-lora", trust_remote_code=True)

# this is the test image
image_url = "https://raw.githubusercontent.com/thaoshibe/relsim/refs/heads/main/anonymous_caption/mam2.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB").resize((448, 448))

####################################################
#
# Uncomment this and put your local image_path here
#
####################################################
# image = Image.open("imag_path").convert("RGB").resize((448, 448))

FIXED_PROMPT = '''
    You are given a single image.
    Carefully analyze it to understand its underlying logic, layout, structure, or creative concept. Then generate a single, reusable anonymous caption that could describe any image following the same concept.
    The caption must:
    - Fully capture the general logic or analogy of the image.
    - Include placeholders (e.g., {Object}, {Word}, {Character}, {Meaning}, {Color}, etc.) wherever variations can occur.
    - Be concise and standalone.
    Important: Only output the anonymous caption. Do not provide any explanations or additional text.
'''
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": FIXED_PROMPT}
    ]
}]

# Process and generate
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info([messages])
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=128, temperature=0.7, top_p=0.9)
    caption = processor.decode(output[0][len(inputs.input_ids[0]):], skip_special_tokens=True)

print(f"Anonymous caption: {caption}")

The result of above code should be as follows:

---

For more details, or training code, data, etc. Please visit: https://github.com/thaoshibe/relsim.

Thank you!!

Citation: arxiv.org/abs/2512.07833

@misc{nguyen2025relationalvisualsimilarity,
      title={Relational Visual Similarity}, 
      author={Thao Nguyen and Sicheng Mo and Krishna Kumar Singh and Yilin Wang and Jing Shi and Nicholas Kolkin and Eli Shechtman and Yong Jae Lee and Yuheng Li},
      year={2025},
      eprint={2512.07833},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.07833}, 
}