meme-template-recovery-clip

Meme Template Recovery Model (CLIP Frozen + Linear Head)

This model projects meme images into a $256$-dimensional metric embedding space where images originating from the same base meme template cluster closely together, and images from different templates are pushed far apart.

It is designed for matching macro meme instances back to their original blank meme backgrounds.

Model Details

• Backbone Status: Frozen pre-trained CLIP vision transformer (openai/clip-vit-base-patch32).
• Projection Head: Linear projection layer mapping pooler output ($768$-D) to $256$-D metric space.
• Normalisation: L2 normalisation (making Euclidean distance equivalent to Cosine similarity).
• Training Loss: Batch-Hard Triplet Loss with online semi-hard negative mining.

Evaluation Results

Evaluated on the held-out open-set test split of unseen Know Your Meme Knowledge Base (KYMKB) templates:

How to Run

Installation

Ensure you have the required dependencies installed:

pip install torch torchvision transformers huggingface_hub pillow

Python Inference Example

Download model.py and infer.py from this repository, and execute inference:

from infer import load_meme_model, get_image_embedding

# Load model and device
model, device = load_meme_model(repo_id="npnkhoi/meme-template-recovery-clip")

# Extract embedding for an image
embedding = get_image_embedding(model, "path/to/your/query_meme.jpg", device)
print("Successfully extracted L2-normalized 256-D embedding:", embedding.shape)